Smoke test plan: what's testable on main today

[harshitsinghbhandari]

This is a manual smoke-test runbook for what is actually shipped on main today — the lanes the rewrite already exposes end-to-end (daemon, projects, sessions, spawn, SCM observer, agent activity hooks, PR claim, reaper, CDC, HTTP API). It is intended as a 1–3 hour walk-through before more wiring lands, so we can catch lane-level regressions while the surface is still small enough to cover by hand.

Anyone is welcome to walk a section, report findings as a reply (what worked, what broke, what was unclear), or call out a capability that should have been listed but isn't.

---

AO smoke-test checklist — what's testable on main today

A manual runbook for validating the AO rewrite end-to-end. Walk the lanes in order — earlier lanes set up state used later. Target wall-clock: 1–3 hours.

CLI shape was captured against the current main (commit b13f413); flag names are copy-pasteable.

---

0. One-time prerequisites

• macOS or Linux. Windows is unsupported (zellij not available).
• go ≥ 1.22, git ≥ 2.25.0 (worktree-capable), zellij ≥ the version pinned in backend/internal/adapters/runtime/zellij, gh ≥ 2.0.
• gh auth login complete, OR AO_GITHUB_TOKEN / GITHUB_TOKEN exported (needed for SCM observer + claim-pr lanes).
• A test GitHub repo you control with:
  • At least one open PR whose CI is currently red (needed for SCM observer nudge).
  • At least one open PR whose CI is green and has at least one unresolved review thread (needed for review-thread/dedup tests).
  • origin remote points at github.com.
• At least one agent harness binary on PATH that you actually want to drive: claude, codex, or opencode.

Build the binary once

cd backend && go build -o /tmp/ao_smoke ./cmd/ao
/tmp/ao_smoke version

Isolated data dir (mirrors the issue-#89 recipe)

export AO_DATA_DIR=/tmp/ao-smoke-data
export AO_RUN_FILE=/tmp/ao-smoke-running.json
export AO_PORT=4101
mkdir -p "$AO_DATA_DIR"
unset AO_AGENT   # leave at the default unless explicitly testing AO_AGENT

Keep this env exported in every shell you use. Use the same /tmp/ao_smoke binary throughout so the daemon and CLI agree on the schema.

---

1. Daemon control

1.1 ao start

/tmp/ao_smoke start --json --log-file /tmp/ao-smoke.log

• Expect: JSON {state: "ready", port: 4101, pid: <int>}.
• Verify: cat /tmp/ao-smoke-running.json shows the same PID; curl -fsS http://127.0.0.1:4101/healthz returns {"status":"ok"...}; curl -fsS http://127.0.0.1:4101/readyz returns {"status":"ready"...}.

1.2 ao status

/tmp/ao_smoke status --json

• Expect: state: "ready", port matches, no error string.
• Verify: identical PID to running.json.

1.3 ao doctor

/tmp/ao_smoke doctor
/tmp/ao_smoke doctor --json

• Expect per runDoctor() (backend/internal/cli/doctor.go):
  • Core: config, data-dir, data-dir-write, sqlite, daemon — all PASS.
  • Tools: git PASS (≥ 2.25.0), zellij PASS.
  • Agent harnesses: claude-code and codex — PASS if installed, WARN if not.
  • GitHub: github-token PASS if gh auth login or AO_GITHUB_TOKEN resolves to a valid token; WARN if neither present.
• Independently exercise each check by removing the dep and re-running:
  • PATH=/usr/bin:/bin /tmp/ao_smoke doctor (strip git/zellij) → expect FAIL on git, WARN on zellij.
  • AO_GITHUB_TOKEN= GITHUB_TOKEN= PATH=/usr/bin:/bin /tmp/ao_smoke doctor → expect WARN on github-token with a "no GitHub token found" message.
  • chmod -w "$AO_DATA_DIR" (then revert) → expect FAIL on data-dir-write.
• Doctor exits non-zero iff there's at least one FAIL.

1.4 ao version

/tmp/ao_smoke version

• Expect: a non-empty version line. Also /tmp/ao_smoke --version works.

1.5 ao stop

/tmp/ao_smoke stop --json

• Expect: JSON success, daemon exits, PID gone, /tmp/ao-smoke-running.json is absent or marked stopped.
• Verify: kill -0 <pid> returns 1; ao status reports not_running.

Restart with /tmp/ao_smoke start --json --log-file /tmp/ao-smoke.log before continuing.

---

2. Project management

2.1 ao project add

Pre-stage two git repos on disk: a clean clone of your test GH repo (/tmp/ao-smoke-repo) and a throwaway local-only repo (/tmp/ao-smoke-local, git init then commit a file).

/tmp/ao_smoke project add --path /tmp/ao-smoke-repo --name "smoke gh"
/tmp/ao_smoke project add --path /tmp/ao-smoke-local --name "smoke local"

• Expect: JSON success with a derived project id (kebab from path) for each.
• Verify the repo origin is captured at add time for the GitHub repo. The wire field is repo (mapped from the internal RepoOriginURL column by projectFromRow):

  /tmp/ao_smoke project get <gh-project-id> --json | jq .project.repo

  Should be the https://github.com/<owner>/<repo>.git URL. For the local-only repo it will be "".
• Adding the same path twice should fail with a duplicate-path error.

2.2 ao project ls

/tmp/ao_smoke project ls
/tmp/ao_smoke project ls --json

• Expect: Both projects listed with id, name, path.

2.3 ao project get

/tmp/ao_smoke project get <gh-project-id>

• Expect: project record echoed. With --json you get the full record including repoOriginURL.
• Unknown id returns 404 / nonzero exit.

2.4 ao project rm

Defer until teardown — sessions in later lanes need these projects.

---

3. Spawn / session lifecycle

3.1 ao spawn (worker)

/tmp/ao_smoke spawn \
  --project <gh-project-id> \
  --harness claude-code \
  --prompt "smoke test: just print pwd and exit" \
  --branch ao/smoke-1

• Expect: a session id printed, e.g. ws-01. A new worktree at ~/.agent-orchestrator/projects/<project>/worktrees/<session-id>/ exists and is on branch ao/smoke-1.
• Verify:
  • ls $AO_DATA_DIR/ao.db exists.
  • /tmp/ao_smoke session ls --json shows the session with kind: "worker" and state: "active" (or whichever the lifecycle reduces to once the runtime is up).
  • A zellij session exists: ZELLIJ_SOCKET_DIR=/tmp/ao-zellij-$(id -u) zellij list-sessions — the AO session should be listed.

3.2 --agent / --harness selection

Top-level help shows --harness is also documented as --agent. Spawn one of each:

/tmp/ao_smoke spawn --project <gh-project-id> --harness codex   --prompt "noop" --branch ao/smoke-codex
/tmp/ao_smoke spawn --project <gh-project-id> --harness opencode --prompt "noop" --branch ao/smoke-opencode

• Expect: each session boots with the requested harness (visible in session get JSON's harness field).
• Alternative selection path: AO_AGENT=opencode /tmp/ao_smoke spawn --project <gh-project-id> --prompt "noop" --branch ao/smoke-env. The daemon's default agent comes from AO_AGENT (config.go); confirm the spawned session reports harness=opencode.

3.3 Agent registry coverage

Confirm the registry exposes the eight adapters added by #119 and #145 (claude-code, codex, opencode, grok, cursor, qwen, copilot, kimi). The four newest don't ship hook installers; spawning is enough to confirm the registry accepts them.

Binary-freshness precondition. Migration 0007_allow_implemented_harnesses.sql widens the sessions.harness CHECK constraint to include the new harnesses. If /tmp/ao_smoke was built before #119 landed, migration 0007 isn't embedded and all five spawns below fail with INTERNAL_ERROR in <1ms (the daemon log shows only the request line — see "Known issues"). Verify before running the loop:

sqlite3 "$AO_DATA_DIR/ao.db" 'SELECT max(version_id) FROM goose_db_version;'   # must be ≥ 7

If it's 6, rebuild and restart:

/tmp/ao_smoke stop
(cd <worktree>/backend && go build -o /tmp/ao_smoke ./cmd/ao)
/tmp/ao_smoke start --json --log-file /tmp/ao-smoke.log

for h in grok cursor qwen copilot kimi; do
  /tmp/ao_smoke spawn --project <gh-project-id> --harness $h --prompt "noop" --branch ao/smoke-$h
done

• Expect: spawn succeeds (or errors clearly on a missing binary, not on "unknown harness").

3.4 ao session ls

/tmp/ao_smoke session ls
/tmp/ao_smoke session ls --json
/tmp/ao_smoke session ls -p <gh-project-id>
/tmp/ao_smoke session ls -a                # includes orchestrators
/tmp/ao_smoke session ls --include-terminated

• Expect: rows match what you've spawned; project filter narrows to one project; --include-terminated adds any sessions you killed.

3.5 ao session get

/tmp/ao_smoke session get <session-id> --json

• Expect: full session record. Note the activity / state fields — used in lane §8.
• -p <project> is supported but optional.

3.6 ao session rename

/tmp/ao_smoke session rename <session-id> "renamed-during-smoke"

• Verify: session get --json shows updated displayName. HTTP path is PATCH /api/v1/sessions/{id}.

3.7 ao session kill

/tmp/ao_smoke session kill <session-id>

• Expect: success; session moves to terminated.
• Verify: session ls no longer shows it without --include-terminated; zellij session has been destroyed; the worktree directory is preserved (cleanup is a separate step).

3.8 ao session restore

/tmp/ao_smoke session restore <terminated-session-id>

• Expect: the session is re-launched with a new runtime; session get shows it active again.

3.9 ao session cleanup

/tmp/ao_smoke session cleanup -y
/tmp/ao_smoke session cleanup -p <gh-project-id> -y

• Expect: terminated sessions with clean worktrees are reclaimed; dirty worktrees are skipped (note printed). The session row is removed and the worktree directory is deleted.
• Verify with git -C ~/.agent-orchestrator/projects/<project> worktree list that pruned ones are gone.

---

4. Orchestrator sessions

4.1 Spawn via API (no CLI shortcut for spawn)

curl -fsS -X POST http://127.0.0.1:4101/api/v1/orchestrators \
  -H 'Content-Type: application/json' \
  -d '{"projectId":"<gh-project-id>","clean":false}' | jq

• Expect: 201 with {orchestrator:{id, projectId}}.

4.2 ao orchestrator ls

/tmp/ao_smoke orchestrator ls
/tmp/ao_smoke orchestrator ls --json

• Expect: the newly spawned orchestrator is listed.
• Verify the API mirrors: GET /api/v1/orchestrators returns the same set; GET /api/v1/orchestrators/{id} returns one; querying with a non-orchestrator id returns 404 (SESSION_NOT_FOUND).

---

5. Inter-session messaging (ao send)

5.1 Plain send

/tmp/ao_smoke send --session <session-id> --message "hello from CLI"

• Expect: success; the agent pane (attach via zellij attach <session>) shows the message piped to the harness.

5.2 PR #85 sender-prefix behavior

The CLI prepends [from <sender>] when AO_SESSION_ID is set (see backend/internal/cli/send.go:46):

AO_SESSION_ID=aa-orchestrator /tmp/ao_smoke send --session <session-id> --message "ping"

• Expect: the agent receives [from aa-orchestrator] ping. Confirm in the pane.

5.3 Validation

• Empty --message → 400 with MESSAGE_REQUIRED.
• Message > 4096 chars → 400 with MESSAGE_TOO_LONG.
• Unknown session id → 404.
• Control chars (non-newline/tab/cr) are stripped (stripUnsafeControlChars).

---

6. SCM observer (issues #76 + #108 + #114 + #115)

6.1 Pre-conditions

• A GH project is registered (lane §2). repoOriginURL is non-empty in project get --json.
• The project has at least one open PR with red CI.
• A token is available via AO_GITHUB_TOKEN, GITHUB_TOKEN, or gh auth token.
• Spawn a session whose branch matches the PR's head branch (so the observer can attach the PR to the session). Either:
  • Spawn with --branch <pr-head-branch>, or
  • Use --claim-pr <pr-url> to immediately attach (lane §9).

6.2 Discovery + initial nudge

Wait ≥ 30s (default tick = DefaultTickInterval in backend/internal/observe/scm/observer.go:25).

• Verify the PR is now tracked:

  /tmp/ao_smoke session get <session-id> --json | jq '.session.pr // .session.prs'
  curl -fsS "http://127.0.0.1:4101/api/v1/sessions/<session-id>/pr" | jq

• Verify a nudge landed in the worker's pane exactly once (attach with zellij attach): a CI-failing observation should send a single CI-failure message to the agent.

6.3 No-token graceful no-op

Stop the daemon, unset every token, restart:

/tmp/ao_smoke stop
unset AO_GITHUB_TOKEN GITHUB_TOKEN
mv ~/.config/gh ~/.config/gh.bak 2>/dev/null
/tmp/ao_smoke start --json

• Expect: daemon log shows scm observer disabled: provider credentials unavailable and observer loop becomes a no-op (o.disabled = true). No nudges fire. Restore the token / gh config when done.

6.4 Dedup across restart

With a live token and a still-red PR:

1. Wait for first nudge.
2. /tmp/ao_smoke stop → /tmp/ao_smoke start.
3. Wait one tick.

• Expect: no second duplicate nudge for the same CI failure fingerprint (the dedup signature persisted in SQLite via the semantic hash columns CIHash / MetadataHash / ReviewHash).

6.5 RepoOriginURL backfill

For the local-only project you registered without origin (lane §2), repoOriginURL is empty. Add an origin remote pointing at your GH repo, then spawn a worker on it. The observer's discoverSubjects runs git remote get-url origin and backfills + persists the URL (observer.go:451-459).

• Verify: project get <local-project-id> --json | jq .project.repo now matches the configured remote.

6.6 Review-thread polling

Pick a PR with at least one unresolved review thread.

• Verify: after one tick (or two minutes per DefaultReviewInterval), threads appear via GET /api/v1/sessions/<id>/pr (reviewComments).
• Resolve the thread on GitHub; after the next tick the thread state in the DB shows resolved. Confirm via the CDC log (§10) — a pr_review_thread_resolved event lands.

---

7. Agent activity hooks (issue #113, #119)

7.1 Hook config installed in workspace

Spawn a fresh claude-code worker, then in the worktree:

ls -la <worktree>/.claude/settings.local.json
cat <worktree>/.claude/settings.local.json | jq .hooks

• Expect: hook config installed by backend/internal/adapters/agent/claudecode/hooks.go. Entries point at ao hooks claudecode <event>.

Spawn a codex worker; expect <worktree>/.codex/hooks.json with similar entries (backend/internal/adapters/agent/codex/hooks.go). For cursor, expect <worktree>/.cursor/hooks.json. opencode and grok have no native hook config — note "no-op" in your run log.

7.2 Activity transitions land in the daemon

From inside an active session, trigger the hook manually (the harness will do this naturally too):

AO_SESSION_ID=<session-id> /tmp/ao_smoke hooks claudecode SessionStart < /dev/null
AO_SESSION_ID=<session-id> /tmp/ao_smoke hooks claudecode UserPromptSubmit < /dev/null
AO_SESSION_ID=<session-id> /tmp/ao_smoke hooks claudecode Stop < /dev/null

• Expect: each invocation exits 0 and POSTs to /api/v1/sessions/{id}/activity with one of active|idle|waiting_input|exited.
• Verify:

  /tmp/ao_smoke session get <session-id> --json | jq '.session.activity'

  shows the latest state; sqlite3 $AO_DATA_DIR/ao.db 'select type,payload from change_log order by seq desc limit 5;' shows a recent session_updated event reflecting the activity column.

---

8. PR claim (#101)

8.1 ao session claim-pr

Spawn a worker; in parallel, you have an open PR on the test repo.

/tmp/ao_smoke session claim-pr <session-id> <pr-url-or-number> --json

• Expect: success; response includes branchChanged and takenOverFrom. The worktree's branch may flip to the PR's head branch (branchChanged: true).
• Verify: session get --json shows the PR attached; git -C <worktree> rev-parse --abbrev-ref HEAD matches PR head branch.

8.2 ao spawn --claim-pr (single-step)

/tmp/ao_smoke spawn --project <gh-project-id> --harness claude-code \
  --claim-pr <pr-url> --prompt "fix this PR"

• Expect: a session is spawned and the PR is claimed atomically.
• Failure rollback: try a non-existent PR url:

  /tmp/ao_smoke spawn --project <gh-project-id> --harness claude-code \
    --claim-pr https://github.com/<owner>/<repo>/pull/999999 --prompt "x"

  • Expect non-zero exit and no orphan session row — confirm session ls --include-terminated does not show a leftover. The orchestrator-spawn path rolls back the half-spawned session.

8.3 --no-takeover

With a PR already claimed by an active session, run claim from a different session without --no-takeover and expect takeover; rerun with --no-takeover and expect a PR_CLAIMED_BY_ACTIVE_SESSION 409 (see writeSessionPRError).

---

9. Reaper

9.1 External kill

Spawn a session. From a separate shell:

ZELLIJ_SOCKET_DIR=/tmp/ao-zellij-$(id -u) zellij kill-session <ao-session-name>

• Expect: within one reaper tick, session ls --include-terminated shows the session terminated with a reaper-attributed reason; session get --json shows the runtime gone. The reaper transitions the row without racing the activity hook (lane §7's single lifecycle.Manager guarantee — see sessions.go:260-263).

---

10. CDC events

10.1 change_log table directly

sqlite3 $AO_DATA_DIR/ao.db \
  'select seq, type, datetime(ts/1e9,"unixepoch") from change_log order by seq desc limit 20;'

• Expect to see, depending on what you've done, rows of these types (from backend/internal/cdc/event.go):
  • session_created, session_updated
  • pr_created, pr_updated, pr_check_recorded, pr_session_changed
  • pr_review_thread_added, pr_review_thread_resolved

10.2 SSE replay stream

curl -N "http://127.0.0.1:4101/api/v1/events?after=0" | head -40

• Expect: a text/event-stream response with the same events, in order, then live updates as you trigger more actions. The Last-Event-ID / ?after= cursor is honored (issue feat(cdc): add SSE event stream replay #106).

---

11. HTTP API surface

Verify each route accepts the documented verbs. Source of truth: backend/internal/httpd/controllers/{projects,sessions,prs}.go and backend/internal/httpd/{api,router,events}.go. OpenAPI: GET /api/v1/openapi.yaml.

Route	Verbs	Notes
/healthz	GET	liveness; always 200 once process is up
/readyz	GET	readiness; 200 once dependencies are wired
/shutdown	POST	graceful daemon shutdown
/api/v1/openapi.yaml	GET	self-hosted OpenAPI doc
/api/v1/projects	GET, POST	list / add
/api/v1/projects/{id}	GET, DELETE	fetch / remove
/api/v1/sessions	GET, POST	list / spawn (projectId required)
/api/v1/sessions/cleanup	POST	reclaim terminated, ?project= filter
/api/v1/sessions/{id}	GET, PATCH	fetch / rename (displayName required)
/api/v1/sessions/{id}/pr	GET	session PR facts
/api/v1/sessions/{id}/pr/claim	POST	claim PR ({pr, allowTakeover})
/api/v1/sessions/{id}/restore	POST	relaunch terminated session
/api/v1/sessions/{id}/kill	POST	terminate session
/api/v1/sessions/{id}/send	POST	send message to session
/api/v1/sessions/{id}/activity	POST	activity hook ingress (state ∈ active/idle/waiting_input/exited)
/api/v1/orchestrators	GET, POST	list / spawn orchestrator
/api/v1/orchestrators/{id}	GET	fetch orchestrator
/api/v1/prs/{id}/merge	POST	trigger merge action
/api/v1/prs/{id}/resolve-comments	POST	resolve threads ({commentIds} optional)
/api/v1/events	GET	SSE CDC stream (?after=<seq> or Last-Event-ID)
/mux	GET (WebSocket)	terminal multiplexer

Smoke them with curl for OPTIONS/wrong verb on a couple to confirm the locked envelope: e.g. curl -X PUT http://127.0.0.1:4101/api/v1/projects/abc → 405 METHOD_NOT_ALLOWED; unknown route → 404 ROUTE_NOT_FOUND, both JSON.

---

Teardown

/tmp/ao_smoke session cleanup -y
/tmp/ao_smoke project rm <gh-project-id> -y
/tmp/ao_smoke project rm <local-project-id> -y
/tmp/ao_smoke stop --json

# leftover zellij sessions, if any
ZELLIJ_SOCKET_DIR=/tmp/ao-zellij-$(id -u) zellij list-sessions \
  | awk '{print $1}' | xargs -I{} zellij kill-session {} 2>/dev/null

rm -rf /tmp/ao-smoke-data /tmp/ao-smoke-running.json /tmp/ao-smoke.log
rm -rf /tmp/ao-smoke-repo /tmp/ao-smoke-local
rm -rf ~/.agent-orchestrator/projects/<gh-project-id> ~/.agent-orchestrator/projects/<local-project-id>

---

Known issues that will distort tests

• Worker orchestrator smoke test: message flow works, CLI PATH and zellij test gaps remain #89 — macOS zellij socket-path-length: zellij's unix-socket path budget is ~103 bytes. AO works around this by anchoring the socket dir at /tmp/ao-zellij-<uid> (zellij.DefaultSocketDir). If you override ZELLIJ_SOCKET_DIR to anything deep (e.g. inside $TMPDIR), spawns fail with "session name must be less than 0 characters." Symptom: Create runtime errors at spawn time on macOS.
• SCM observer no-token graceful no-op: when no token is resolvable, checkCredentials flips the observer to disabled for the lifetime of the process. Subsequent token availability is not re-detected — restart the daemon after fixing the token, otherwise observer-dependent lanes (§6, §10's pr_* events) will silently produce nothing.
• AO CI-failure notifier should suppress pings when jobs never ran (billing/quota/admin block) #82 — billing-block notifier suppression (not yet shipped): until the billing-block CI-failure suppression lands, repeated red CI on the same PR may produce a single nudge but the dedup behavior is sensitive to which check fingerprint changes — distinguish "no duplicate" (intended) from "stuck dedup" by changing the failure (push a fresh commit) and confirming a fresh nudge fires.
• Reaper tick latency: section §9 may take up to ~10–30s to react depending on the reaper cadence — don't conclude a failure inside 5s.
• INTERNAL_ERROR with no daemon-log reason — envelope.WriteError returns INTERNAL_ERROR for any non-apierr.Error and does not log the wrapped err. If a spawn/kill/etc. returns 500 in <1ms and the log only shows the HTTP request line, suspect a SQLite CHECK-constraint violation (e.g. stale migration on sessions.harness — see §3.3 binary-freshness check) or another low-level store error. Diagnose via sqlite3 "$AO_DATA_DIR/ao.db" 'SELECT max(version_id) FROM goose_db_version;' and comparing to backend/internal/storage/sqlite/migrations/.
• ao hooks exits silently if AO_SESSION_ID is unset — only run from within an agent process (or set it manually as shown in §7.2).
• Cleanup skips dirty worktrees: ao session cleanup is a no-op on a worktree with uncommitted changes. Either commit/discard inside the worktree or rm -rf it manually.

---

Not yet testable (designed but not on main)

• Tracker observer (Tracker observer loop: poll + ApplyTrackerFacts into LCM #35) — provider-neutral tracker analogue of the SCM observer. Not wired.
• Frontend dashboard — frontend/ is a placeholder; no UI to walk through.
• Notifier rebuild — notifier-openclaw/slack/webhook log lines at daemon startup are no-ops until reconfigured.
• Excluded from Port missing CLI commands from original AO (read + lifecycle surface) #90 — ao plugin, ao events, ao update, ao migrate-storage are intentionally absent on main; don't test them.

---

How to participate

Pick a lane (§1 through §11). Walk the steps top to bottom. Reply on this thread with:

• What worked as described.
• What broke, with the failing command's stderr and (if relevant) ao status --json / ao doctor --json output.
• What was unclear — wording, missing prereq, ambiguous expected outcome.

Mention which OS, zellij version, and ao version you ran against.

---

Related

• Worker orchestrator smoke test: message flow works, CLI PATH and zellij test gaps remain #89 — smoke test gaps already known (macOS zellij socket-path limit, and similar footguns we already track)
• AO CI-failure notifier should suppress pings when jobs never ran (billing/quota/admin block) #82 — billing-block notifier suppression; a prerequisite for stable PR-engine smoke testing
• Milestone: rewrite

[harshitsinghbhandari]

Walked lanes 1, 4, 11 on an isolated daemon — port 4201, AO_DATA_DIR=/tmp/ao-smoke-A, binary built from 9dedae9 (version reports dev), Darwin 25.4.0, zellij 0.44.3, git 2.50.1.

What worked

Lane 1 — Daemon control

• 1.1 ao start --json → ready, pid+port reported, running.json written; /healthz and /readyz both 200 with expected JSON.
• 1.2 ao status --json → state: "ready", pid matches running.json.
• 1.3 ao doctor baseline → all PASS. Deliberate exercises:
  • chmod -w "$AO_DATA_DIR" → FAIL on data-dir-write with open .../.ao-doctor-write-...: permission denied; exit 1.
  • AO_GITHUB_TOKEN= GITHUB_TOKEN= PATH=/usr/bin:/bin → WARN no GitHub token found (set AO_GITHUB_TOKEN/GITHUB_TOKEN or run gh auth login).
  • Stripped PATH → WARN on zellij / claude-code / codex (as expected).
• 1.4 ao version and ao --version both non-empty.
• 1.5 ao stop --json → JSON ack, pid gone, running.json removed; subsequent ao status reports stopped.

Lane 4 — Orchestrator sessions (partial — see "broke" below)

• Local-only repo registered; ao orchestrator ls / --json enumerates sessions; JSON shape is {data: [{id, projectId, role, status, isTerminated, lastActivityAt, createdAt, updatedAt}, ...]}.
• GET /api/v1/orchestrators/{id} returns full payload for a valid id and 404 SESSION_NOT_FOUND for unknown ids.

Lane 11 — HTTP API surface

• Liveness/readiness/openapi: /healthz, /readyz, /api/v1/openapi.yaml all 200.
• Projects: GET /api/v1/projects 200; duplicate POST → 409 PATH_ALREADY_REGISTERED with a suggestedProjectId; GET /api/v1/projects/{id} 200; DELETE unknown → 404 PROJECT_NOT_FOUND.
• Sessions: GET 200; POST w/o body → 400 PROJECT_ID_REQUIRED; POST /api/v1/sessions/cleanup?project=... → 200 {ok:true, cleaned:[]}; GET .../{unknown} → 404 SESSION_NOT_FOUND; PATCH w/o body → 400 DISPLAY_NAME_REQUIRED; PATCH {displayName:...} → 200; GET .../pr → 200 {prs:[]}; POST .../pr/claim on terminated session → 409 SESSION_TERMINATED; POST .../kill on incomplete handle → 409 SESSION_INCOMPLETE_HANDLE; POST .../send empty → 400 MESSAGE_REQUIRED; >4096 chars → 400 MESSAGE_TOO_LONG; POST .../activity {state:"active"} → 200; bogus state → 400 INVALID_ACTIVITY_STATE.
• PRs: POST /api/v1/prs/{id}/merge and .../resolve-comments → 501 NOT_IMPLEMENTED (echo OpenAPI spec).
• Events: /api/v1/events?after=0 → text/event-stream, Cache-Control: no-cache, X-Accel-Buffering: no, chunked; Last-Event-ID: 4 correctly resumed from seq 5.
• Mux: GET /mux without Upgrade → 426 WebSocket protocol violation (expected; route wired).
• Negative checks: PUT /api/v1/projects/abc → 405 METHOD_NOT_ALLOWED; GET /api/v1/does-not-exist → 404 ROUTE_NOT_FOUND; GET /shutdown → 405; DELETE /sessions/abc/pr → 405. All errors use the locked envelope.

What broke

Lane 4.1 — POST /api/v1/orchestrators always returns 500. Every spawn attempt — valid project, clean:true/false, even unknown projectId — returned INTERNAL_ERROR with no log detail beyond the request line. Matches the "Known issues / INTERNAL_ERROR with no daemon-log reason" footgun exactly.

$ curl -sS -X POST http://127.0.0.1:4201/api/v1/orchestrators \
    -H 'Content-Type: application/json' -d '{"projectId":"ao-smoke-a-local","clean":false}'
{"error":"internal","code":"INTERNAL_ERROR","message":"Internal server error","requestId":"..."}
HTTP=500

SQLite migration is at version 7 (so not the §3.3 stale-migration footgun), and the harness CHECK allows '' for orchestrators. Each failed POST nevertheless leaves a terminated orchestrator row in sessions — including one for projectId:"does-not-exist", which is itself a bug (should validate project before inserting the session row).

Lane 11 — POST /api/v1/sessions/{id}/restore returns opaque 500 for a terminated / half-spawned session instead of a clean envelope.

$ curl -sS -X POST http://127.0.0.1:4201/api/v1/sessions/ao-smoke-a-local-1/restore
{"error":"internal","code":"INTERNAL_ERROR","message":"Internal server error","requestId":"..."}
HTTP=500

Same opaque-500 pattern (no log detail). For comparison, .../kill on the same row returns a clean 409 SESSION_INCOMPLETE_HANDLE.

Lane 1.3 — PATH=/usr/bin:/bin /tmp/ao_smoke doctor does NOT FAIL on git as the runbook predicts. On macOS /usr/bin/git exists (Apple-shipped git 2.50.1), so doctor still finds it; the check passes. Need a different way to force missing git (e.g. PATH=/nonexistent).

What was unclear

• §1.5 says "ao status reports not_running" but the actual value is "stopped" in both human and JSON output. Doc or daemon should align.
• §4.1's expected status is 201 with {orchestrator:{id, projectId}} — on this build it consistently returns 500. No indication whether POST /api/v1/orchestrators is supposed to work on 9dedae9 at all, or whether it needs extra prereqs (a harness arg? gh auth? non-empty repo origin?). The opaque 500 makes this very hard to diagnose without source-diving.
• §11 table lists /api/v1/prs/{id}/merge and .../resolve-comments without flagging that they currently return 501 NOT_IMPLEMENTED. Worth a "shipped: stub" annotation.
• §1.3's PATH=/usr/bin:/bin strip trick doesn't strip git on macOS — doc should suggest an empty/garbage PATH if you want to force git FAIL.
• ao orchestrator ls (human format) prints the status column twice — [terminated] terminated. Cosmetic.

[harshitsinghbhandari]

Walked lanes 2, 3, 5, 7, 9, 10 on an isolated daemon — port 4202, AO_DATA_DIR=/tmp/ao-smoke-B, binary built from 9dedae9 (version dev), Darwin 25.4.0, zellij 0.44.3. goose_db_version = 7 (§3.3 binary-freshness precondition satisfied).

What worked

• Lane 2 (projects) — project add registered both repos; ids derived as ao-smoke-b-repo and ao-smoke-b-local. Duplicate path returned PATH_ALREADY_REGISTERED. project ls --json listed both. project get --json returned full record; repo populated for the GH repo (https://github.com/aoagents/ReverbCode.git) and empty "" for the local. Unknown id returned PROJECT_NOT_FOUND with nonzero exit.
• Lane 3 (sessions / spawn) — Spawned 9 sessions across claude-code, grok, cursor, qwen, copilot, kimi, codex, opencode, plus an AO_AGENT=opencode env-spawn. Worktrees created under /tmp/ao-smoke-B/worktrees/ao-smoke-b-repo/<sid>/, zellij sessions live. Registry log confirmed all 23 adapters loaded. session ls / -p / --include-terminated filter correctly. session rename updated displayName. session kill → terminated; session restore brought it back to idle.
• Lane 5 (ao send) — Plain send and AO_SESSION_ID=… prefix-send both succeeded. Validation: empty --message rejected client-side as usage: --message is required (exit 2); direct API POST /sessions/{id}/send with {"message":""} returns 400 MESSAGE_REQUIRED. >4096 chars → MESSAGE_TOO_LONG (exit 1). Unknown session → SESSION_NOT_FOUND.
• Lane 7 (activity hooks) — .claude/settings.local.json installed in claude-code worktree with all 5 hook entries (Notification, SessionEnd, SessionStart, Stop, UserPromptSubmit). .codex/hooks.json and .cursor/hooks.json present in their respective worktrees. opencode and grok worktrees had no hook files (expected no-op). ao hooks claude-code session-start|user-prompt-submit|stop all exited 0; session activity updated; CDC session_updated rows landed (seq 12/13).
• Lane 9 (reaper) — zellij kill-session ao-smoke-b-repo-2 triggered reaper transition within ~35s; session moved to terminated / isTerminated:true, dropped from session ls, shown in --include-terminated, and CDC seq 14 logged session_updated with activity:exited, isTerminated:true.
• Lane 10 (CDC) — change_log populated with 14 rows. curl -N /api/v1/events?after=0 streamed text/event-stream with id: / event: / data: triples in seq order, agreeing with the SQLite rows.

What broke / discrepancies

• project add --json is unrecognized. Runbook §2.1 says project add ... --json returns "JSON success with a derived project id." Actual flags are only --id, --name, --path. Command fails with unknown flag: --json (exit 2). Output of the bare command is a plain text line, e.g. registered project ao-smoke-b-repo at /tmp/ao-smoke-B-repo.
• change_log column name mismatch. Runbook §10.1 example SELECT uses type; actual column is event_type. The SELECT in the runbook fails with no such column: type.
• session ls --json envelope is {data, meta}, not {sessions}. Runbook §3.4 says "rows match what you've spawned" without specifying shape, but the implied .sessions[] doesn't exist; correct path is .data[]. Minor doc nit.
• AO_AGENT=opencode … spawn produced a session whose session get --json omits the harness field entirely. Runbook §3.2 says "confirm the spawned session reports harness=opencode." For the env-spawned session the field is absent. Explicit --harness … spawns all show the right harness in session ls --json / session get --json.
• qwen spawn (no binary on PATH) succeeded silently — session in idle, zellij session live, no daemon-log signal that the harness exec failed. Runbook tolerates this ("spawn succeeds OR errors clearly on a missing binary"), but the spirit of the lane is only half-satisfied: the only way to discover qwen is missing is to zellij attach and read the pane.

What was unclear

• Hook command name casing. §7.2 uses ao hooks claudecode <Event> (e.g. SessionStart); installed config and the actual subcommand both accept the agent slug claude-code (kebab) and event names like session-start. Both forms exit 0 — runbook should be updated for consistency.
• session get --json schema variance. Some sessions return a harness key, others omit it (see env-driven spawn above). Unclear whether harness is meant to be always-present.

[harshitsinghbhandari]

Walked lanes 6 (SCM observer) and 8 (PR claim) on an isolated daemon — port 4203, AO_DATA_DIR=/tmp/ao-smoke-C, binary built from 9dedae9 (version dev), Darwin 25.4.0. Throwaway repo used for the GH-state prereqs: harshitsinghbhandari/ao-smoke-149-1780787250 (PR #1 red CI, PR #2 green with one unresolved review thread — left intact for inspection).

What worked

Lane 6 — SCM observer

• 6.2 Discovery + initial nudge: spawned worker with --branch pr-red; within ~30s GET /api/v1/sessions/<id>/pr returned PR chore: scaffold backend/ and frontend/ skeletons for rewrite #1 attached with ci: failing. CDC pr_created + two pr_check_recorded rows landed (the failing fail workflow + a passing CodeRabbit check).
• 6.3 No-token graceful no-op: with AO_GITHUB_TOKEN / GITHUB_TOKEN unset and ~/.config/gh moved aside, daemon started cleanly and no new pr_ events* landed across a full observer cycle.
• 6.4 Dedup across restart: stop then start; after a fresh observer tick, no duplicate pr_check_recorded rows for the same fingerprint.
• 6.5 RepoOriginURL backfill: created /tmp/ao-smoke-local-C git repo, ran project add (project's repo came back ""), then added an origin remote pointing at the GH repo. After spawning a worker and one observer tick, project get --json | jq .project.repo showed the GH URL — backfill works.
• 6.6 Review-thread polling: PR feat(backend): Lifecycle Manager + Session Manager lane #2's unresolved thread showed up as pr_review_thread_added in change_log and reviewComments: true in the PR payload.

Lane 8 — PR claim

• 8.1 session claim-pr: attached PR feat(backend): Lifecycle Manager + Session Manager lane #2 to a session — returned ok: true, PR record populated correctly (ci: passing, reviewComments: true).
• 8.3 Takeover semantics: re-claiming PR feat(backend): Lifecycle Manager + Session Manager lane #2 from a different session succeeded with takenOverFrom: ["..."]. Re-claiming with --no-takeover returned PR_CLAIMED_BY_ACTIVE_SESSION with the suggested-flag hint.

What broke

1. §8.2 rollback is incomplete. ao spawn --claim-pr <bad-url> printed failed to claim PR ... PR_NOT_FOUND ...; rolled back session ...-4, but:

• The session row persists in --include-terminated as status: "terminated", activity.state: "exited". Runbook says "no orphan session row".
• CLI exit code is 0 despite a failure. Same exit-0-on-error pattern also seen on session claim-pr --no-takeover returning 409.

2. §6.6 — pr_review_thread_resolved event never landed. After GraphQL resolveReviewThread succeeded (isResolved: true on GitHub), I polled change_log for ~4 minutes (well past DefaultReviewInterval = 2m). Only the pr_review_thread_added rows existed. The PR record's reviewComments also stayed true. The observer ingests thread-added but does not appear to ingest the resolved transition.

3. §6.3 — missing log line. Spec says: with no token, expect scm observer disabled: provider credentials unavailable in the daemon log. The behavior is correct (no nudges, no events), but the explicit log line never appeared in the no-token daemon log. Either the message has changed wording or it's not being emitted.

4. project get --json field naming. The repo field is populated as expected, but .project.repoOriginURL is always null in the JSON. The runbook references both names — only repo ships on the wire. Worth confirming whether repoOriginURL was renamed and the runbook stale, or whether both should be present.

5. Spawn returns opaque INTERNAL_ERROR (500) when the requested branch isn't fetched locally. Two repros:

• First spawn against the local-only project with --branch ao/local-1 failed 500. After git fetch origin (so remote refs existed), the next spawn worked.
• Spawn against the GH project with --branch pr-green failed 500 while the canonical worktree had pr-green checked out. After switching the canonical worktree to main, the spawn worked.

Daemon log shows only the request line + 500 status; no error message. Caller can't tell whether it's a worktree collision, a missing ref, or something else — yet another instance of the "Known issues / INTERNAL_ERROR with no daemon-log reason" footgun.

What was unclear

• Whether ao spawn --claim-pr <bad-pr> is supposed to delete the orphan session row entirely or just mark it terminated. The runbook wording ("no orphan session row — confirm session ls --include-terminated does not show a leftover") suggests deletion, but current behavior is mark-as-terminated.
• Whether the SCM observer is supposed to emit any log line on the no-token path, or just silently disable.
• Field naming in project get --json: repo vs repoOriginURL.
• The exit-code-0-on-error pattern across spawn --claim-pr <bad> and session claim-pr --no-takeover 409 — by design or bug?