feat: add opt-in codex app-server probe

[e-jung]

Intent

Implement a contained Codex app-server / Codex-managed-worktree spike for firstmate. Add the smallest safe, opt-in probe/scaffold that can inspect the installed Codex app-server command/schema and run stdio-only app-server basics without touching real captain projects or exposing unauthenticated non-loopback listeners. Document the evidence, GUI visibility uncertainty, metadata needed for codex_thread_id/active turn/worktree/archive/read semantics, Codex-managed worktree lifecycle differences, and recommend when a future backend=codex-app should start ship/scout-only before secondmates. Do not register codex-app as a spawn-capable backend or destabilize Herdr/tmux.

What Changed

• Adds bin/fm-codex-appserver-probe.sh, an opt-in, stdio-only probe that inspects the installed Codex app-server command/schema and exercises basic create/read/archive-thread lifecycle without registering codex-app as a spawn-capable backend or destabilizing the existing tmux/herdr/zellij/orca backends.
• Adds docs/codex-app-backend-spike.md recording the spike evidence and recommendations: metadata gaps for thread/turn/worktree/archive semantics, Codex-managed worktree lifecycle differences, and guidance that a future backend=codex-app should start ship/scout-only before secondmates.
• Adds tests/fm-codex-appserver-probe.test.sh covering dry-run, schema lifecycle, and live stdio paths, and fixes node-on-PATH flakiness in tests/fm-session-start.test.sh.

Risk Assessment

✅ Low: The change is a well-bounded, opt-in, standalone experimental probe with safe dry-run defaults, stdio-only live mode, explicit refusal of unauthenticated non-loopback listeners, matching tests, and no modification to any existing firstmate spawn/backend machinery; round-1 issues are fixed and no new material issues remain.

Testing

Ran the two affected test files (fm-codex-appserver-probe, fm-session-start) and the full safety/non-registration matrix end-to-end against a fake codex CLI: default dry-run, help, non-loopback WS refusal, loopback + stdio-only gating, unsupported-listen refusal, schema generation/summary incl. failure exit-code propagation and missing-method detection, live stdio create/read/archive incl. keep-thread and graceful no-rollout archive degradation, and the fm-spawn.sh --backend codex-app rejection proving codex-app is not registered while tmux/herdr/zellij/orca remain. All targeted tests pass; working tree left clean; no UI surface exists for this CLI-only spike so the captured CLI transcripts are the reviewer-facing evidence.

Evidence: Probe test results (fm-codex-appserver-probe.test.sh)

ok - fm-codex-appserver-probe: default invocation is safe dry-run ok - fm-codex-appserver-probe: refuses unauthenticated non-loopback WebSocket listener ok - fm-codex-appserver-probe: schema generation summarizes lifecycle methods ok - fm-codex-appserver-probe: live stdio probe drives create/read/archive protocol path

ok - fm-codex-appserver-probe: default invocation is safe dry-run
ok - fm-codex-appserver-probe: refuses unauthenticated non-loopback WebSocket listener
ok - fm-codex-appserver-probe: schema generation summarizes lifecycle methods
ok - fm-codex-appserver-probe: live stdio probe drives create/read/archive protocol path

Evidence: Default dry-run + help (end-user experience)

## default invocation mode: dry-run would_create_thread: no (pass --create-thread ...) safety: no listener opened; no Codex thread created; no project worktree modified

########## END-USER DEMO 1: default invocation (safe dry-run) ##########
mode: dry-run
would_check: codex --help
would_generate_schema: codex app-server generate-json-schema --out <dir>
would_live_handshake: codex app-server --stdio
would_create_thread: no (pass --create-thread to create/read a throwaway no-turn thread)
safety: no listener opened; no Codex thread created; no project worktree modified

########## DEMO 2: --help ##########
usage: fm-codex-appserver-probe.sh [options]

Safe defaults:
  With no options this is a dry run. It does not start app-server, create
  threads, expose listeners, or touch project worktrees.

Options:
  --dry-run                 Print the probe plan only (default)
  --schema-dir DIR          Generate the installed app-server JSON schema bundle
                            into DIR and summarize lifecycle methods
  --experimental-schema     Include experimental schema methods/fields
  --live-handshake          Start app-server over stdio and run initialize +
                            thread/list only
  --create-thread           With the live stdio probe, create a thread in a temp
                            cwd and read it; archive is best-effort because a
                            no-turn thread may not have a rollout yet
  --keep-thread             Do not archive a created thread; prints codex:// link
  --cwd DIR                 cwd for --create-thread (default: fresh temp dir)
  --model MODEL             model override for --create-thread
  --listen URL              Transport to validate. Live probe supports stdio://
                            only; ws:// non-loopback requires auth flags.
  --ws-auth MODE            capability-token or signed-bearer-token
  --ws-token-file PATH      capability token file for ws auth
  --ws-token-sha256 HEX     capability token verifier for ws auth
  --ws-shared-secret-file PATH
                            signed bearer shared secret file for ws auth
  -h, --help                Show this help

Examples:
  fm-codex-appserver-probe.sh
  fm-codex-appserver-probe.sh --schema-dir /tmp/fm-codex-schema
  fm-codex-appserver-probe.sh --live-handshake
  fm-codex-appserver-probe.sh --live-handshake --create-thread --keep-thread

Evidence: Safety guards: non-loopback WS refused, loopback+stdio-only, unsupported listen

refusing unauthenticated non-loopback WebSocket listener 'ws://0.0.0.0:4500' (exit 1) live probe currently supports stdio:// only (exit 1) unsupported --listen URL: tcp://example.com:1234 (exit 1)

########## DEMO 3: unsafe non-loopback WS refused (without auth) ##########
error: refusing unauthenticated non-loopback WebSocket listener 'ws://0.0.0.0:4500'
exit=1

########## DEMO 4: loopback WS allowed past the auth gate (still refuses live because stdio-only) ##########
error: live probe currently supports stdio:// only; use codex app-server directly for authenticated WebSocket testing
exit=1

########## DEMO 5: unsupported --listen URL refused ##########
error: unsupported --listen URL: tcp://example.com:1234
exit=1

Evidence: codex-app NOT a spawn-capable backend; Herdr/tmux untouched

error: unknown backend 'codex-app' (known: tmux herdr zellij orca) bin/fm-backend.sh:58:FM_BACKEND_SPAWN="tmux herdr zellij orca"

########## DEMO 6: codex-app is NOT a spawn-capable backend ##########
error: unknown backend 'codex-app' (known: tmux herdr zellij orca)
exit=1

########## DEMO 7: known backends list (should NOT include codex-app) ##########
(no codex-app references in backend dispatcher or backends dir)

########## DEMO 8: implemented/spawn-capable backends (should be tmux/herdr/zellij/orca only) ##########
bin/fm-backend.sh:58:FM_BACKEND_SPAWN="tmux herdr zellij orca"
bin/fm-backend.sh:148:  fm_backend_list_contains "$FM_BACKEND_SPAWN" "$name" && return 0
bin/fm-backend.sh:149:  echo "error: backend '$name' does not support task spawning yet (spawn-supported: $FM_BACKEND_SPAWN)" >&2

Evidence: Schema generation + live create/read/archive + keep-thread output

schema_required_lifecycle: present schema_turn_methods: turn/interrupt,turn/start,turn/steer live_initialize: ok created_thread_id: 00000000-0000-4000-8000-000000000001 codex_thread_url: codex://threads/00000000-0000-4000-8000-000000000001 archived_created_thread: yes

########## END-USER DEMO: schema generation + lifecycle summary ##########
codex_cli: codex-cli 0.142.5-test
schema_dir: /tmp/tmp.X1q19yTxA8/schema-out
schema_method_count: 8
schema_required_lifecycle: present
schema_thread_methods: thread/archive,thread/list,thread/read,thread/start
schema_turn_methods: turn/interrupt,turn/start,turn/steer
exit=0

########## END-USER DEMO: live stdio create/read/archive ##########
codex_cli: codex-cli 0.142.5-test
live_initialize: ok
codex_home: /tmp/fake-codex
live_thread_list: ok count=0
created_thread_id: 00000000-0000-4000-8000-000000000001
codex_thread_url: codex://threads/00000000-0000-4000-8000-000000000001
thread_read_after_create: ok includeTurns=true
archived_created_thread: yes
thread_cwd: /tmp/fm-codex-appserver-probe-QfxL2S
notifications_seen: 
exit=0

########## END-USER DEMO: keep-thread prints codex:// link ##########
codex_cli: codex-cli 0.142.5-test
live_initialize: ok
codex_home: /tmp/fake-codex
live_thread_list: ok count=0
created_thread_id: 00000000-0000-4000-8000-000000000001
codex_thread_url: codex://threads/00000000-0000-4000-8000-000000000001
thread_read_after_create: ok includeTurns=true
archived_created_thread: no
thread_cwd: /tmp/fm-codex-appserver-probe-ExwK3S
notifications_seen: 
exit=0

Evidence: Archive no-rollout graceful degradation

archived_created_thread: unavailable-no-rollout

########## archive fails with 'rollout' -> reported as unavailable-no-rollout ##########
codex_cli: codex-cli 0.142.5-test
live_initialize: ok
codex_home: /tmp/fake-codex
live_thread_list: ok count=0
created_thread_id: abc-rollout-test
codex_thread_url: codex://threads/abc-rollout-test
thread_read_after_create: ok includeTurns=true
archived_created_thread: unavailable-no-rollout
thread_cwd: /tmp/fm-codex-appserver-probe-7BljlQ
notifications_seen: 
exit=0

Evidence: Schema exit-code propagation (gen failure exit 3; missing-method exit 2)

schema gen failed -> exit=3 schema_required_lifecycle: missing thread/list,thread/read,thread/archive,... -> exit=2

########## schema generation failure -> nonzero exit propagated ##########
codex_cli: codex-cli 0.142.5-test
boom: schema gen failed
exit=3

########## schema present but missing a required lifecycle method -> summarize fails nonzero ##########
codex_cli: codex-cli 0.142.5-test
schema_dir: /tmp/tmp.FkZfiN4vM3/schema-out2
schema_method_count: 2
schema_required_lifecycle: missing thread/list,thread/read,thread/archive,turn/start,turn/steer,turn/interrupt
schema_thread_methods: thread/start
schema_turn_methods: 
exit=2

- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (30m23s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 3 issues found → auto-fixed ✅

• ⚠️ bin/fm-codex-appserver-probe.sh:351 - Schema-generation failures are masked by exit 0. The script uses only set -u (no set -e), so when codex app-server generate-json-schema (line 351) or summarize_schema (line 352) fails, the exit codes are discarded and execution falls through to the trailing if [ "$LIVE_HANDSHAKE" -eq 1 ] (lines 355-357). When no live handshake runs (or it succeeds), the script exits 0 even though schema generation failed - I confirmed this with a trace: a failing codex+summarize pair yields exit=0. Automated callers/CI relying on the exit code would silently miss schema failures. Propagate the schema command/function exit codes (e.g. capture and return them, or gate the trailing block on success).
• ⚠️ bin/fm-codex-appserver-probe.sh:249 - Graceful-degradation paths hinge on substring-matching external Codex error text: includes("includeTurns") (line 249) and includes("no rollout") (line 260). The spike docs explicitly position this probe as the tool to re-run as Codex versions change, so if Codex rewords these messages the probe will throw instead of reporting the documented unavailable-no-rollout / includeTurns=false fallbacks. Unexpected errors are still surfaced loudly (good), but the known-state handling is version-fragile. Consider matching on error codes if Codex exposes them, lowercasing the comparison, or tolerating any failure for a no-turn thread rather than a literal phrase.
• ℹ️ bin/fm-codex-appserver-probe.sh:54 - The WebSocket loopback/auth validation is advisory only: the live probe unconditionally rejects any non-stdio transport (lines 323, 325), so is_loopback_ws (line 54, whose ws://127.* glob would also accept crafted hosts like ws://127.0.0.1.evil.com) and has_ws_auth_material (which treats a single capability-token field as sufficient) never gate an actual listener. No current security impact, but when a future backend=codex-app wires real ws these loose checks should be tightened (parse the host; require both token + verifier).

🔧 Fix: propagate schema exit codes and harden codex app-server probe error matching
✅ Re-checked - no issues remain.

🔧 **Test** - 1 issue found → auto-fixed ✅

• 🚨 tests failed with exit code 1
• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: fix node-on-PATH test flakiness in fm-session-start tests
✅ Re-checked - no issues remain.

• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• bash tests/fm-codex-appserver-probe.test.sh (4/4 assertions: dry-run default, non-loopback WS refusal, schema lifecycle summary, live stdio create/read/archive)
• bash tests/fm-session-start.test.sh (all session-start composition/ordering/liveness assertions, incl. the lavish-axi MISSING-line determinism fix)
• bash bin/fm-codex-appserver-probe.sh (no args) -> dry-run safety banner
• bash bin/fm-codex-appserver-probe.sh --help -> usage/options/examples
• bash bin/fm-codex-appserver-probe.sh --live-handshake --listen ws://0.0.0.0:4500 -> refuses unauthenticated non-loopback WS (exit 1)
• bash bin/fm-codex-appserver-probe.sh --live-handshake --listen ws://127.0.0.1:4500 -> loopback passes auth gate, still refuses live (stdio-only) (exit 1)
• bash bin/fm-codex-appserver-probe.sh --listen tcp://example.com:1234 -> unsupported listen refused (exit 1)
• schema generation + lifecycle summary against fake codex (8 methods, schema_required_lifecycle: present)
• live stdio create/read/archive against fake codex (live_initialize: ok, created_thread_id, codex_thread_url, archived_created_thread: yes)
• live stdio --create-thread --keep-thread against fake codex (archived_created_thread: no, temp cwd still cleaned up)
• archive-error-with-rollout matching -> archived_created_thread: unavailable-no-rollout (exit 0, graceful degradation)
• schema-generation failure propagates codex exit 3; schema-present-but-missing-lifecycle -> summarize exit 2
• FM_HOME=/tmp/... bash bin/fm-spawn.sh ... --backend codex-app -> rejected as unknown backend (known: tmux herdr zellij orca); grep confirms no codex-app references in bin/fm-backend.sh or bin/backends/
• post-test git status --porcelain clean (no transient artifacts left in worktree)

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.