Reef v2 orchestration control plane and operator UX

[pranavpatilsce]

Summary

This PR lands the Reef v2 orchestration control plane and operator UX.

At a product level, it turns Reef from a thin chat wrapper into an active fleet supervisor with:

• a single vm-tree-backed source of truth
• recursive orchestration across root, lieutenants, agent VMs, swarm workers, and resource VMs
• explicit authority / coordination / synchronization primitives
• durable usage, logs, scheduling, and lineage views
• a chat-first operator UI with persistent fleet world state

This PR also includes the latest runtime and prompt-contract tightening needed to make repo-oriented implementation safer:

• stronger root task startup handling
• hardened GitHub repo preparation
• safer root deployment defaults
• clearer ownership and delegation defaults for non-trivial repo builds

What changed

Fleet state and orchestration model

• unified fleet state under vm-tree
• removed the old registry service from the active Reef surface
• added explicit VM categories:
  • infra_vm
  • lieutenant
  • agent_vm
  • swarm_vm
  • resource_vm
• fixed nested lineage so spawned descendants keep their true parentage instead of flattening to root
• split operational views from history:
  • active-by-default fleet/tree views
  • explicit history-inclusive lineage views
• kept running resource_vm infrastructure visible in active views even when its parent subtree becomes historical

Communication and coordination

• added first-class communication semantics:
  • reef_signal upward
  • reef_command downward
  • reef_peer_signal for same-parent sibling coordination
• enforced hierarchical command authority while allowing bounded peer coordination
• added store coordination primitives:
  • reef_store_list for discovery across namespaced keys
  • reef_store_wait for barriers, waits, and synchronization
• made exact-key waits namespace-transparent across agent-owned keys for cross-agent coordination
• added scheduled orchestration primitives:
  • reef_schedule_check
  • reef_scheduled
  • reef_cancel_scheduled
• removed reminder timers from the active orchestration surface and moved follow-up toward scheduled checks

Runtime and lifecycle behavior

• made lieutenant behavior durable by default instead of disposable-worker by default
• kept root supervision bounded to turns while preserving continuity across turns
• preserved post-mortem access to stopped child logs
• fixed swarm completion surfacing into the main signal plane
• fixed swarm wait semantics so it resolves on active workers instead of hanging on idle siblings
• kept resource_vm protected-by-default and out of token/cost accounting
• added explicit child lifecycle tasking for alive idle agent VMs
• hardened post-task disposition handling and target semantics

Usage, logs, and observability

• added lineage-aware usage rollups across lieutenants, agent VMs, and swarm workers
• ensured stopped descendants still contribute to subtree usage
• kept resource_vm at zero usage
• built a real log browser:
  • keyword search
  • date range search
  • agent filter
  • level filter
  • count and totalCount
• routed the logs panel through the UI auth proxy so browser usage works correctly
• added the persistent reef world state section above the composer
• made that section more live and useful:
  • SSE-assisted refresh
  • polling fallback
  • partial-failure tolerance
  • scrollable cards
  • receiving-card fallback when inbox is empty

Root deployment safety and service runtime

• added a root task startup watchdog with a bounded retry so tasks fail explicitly instead of silently hanging
• hardened reef_git_prepare for real repo work
• made service reload handle symlinked service directories consistently
• improved /services/deploy diagnostics so path and service-root mistakes are legible
• gated Reef-root service deploys behind explicit control-plane intent

AGENTS and skills

• slimmed AGENTS.md down to the environment contract and moved procedures into skills
• added/updated skills for:
  • decomposition
  • code delivery
  • GitHub repo preparation
  • app deployment outside Reef root
  • root supervision
  • reporting/checkpointing
  • service creation
• tightened repo-build behavior so:
  • repo-local guidance is read early
  • product/application work defaults outside Reef root
  • non-trivial repo builds assign ownership earlier
  • root is less likely to silently reclaim child-owned work

Why this matters

Before this change, Reef had split control-plane concepts and too much orchestration state lived in fragile conventions.

After this change:

• fleet state is legible
• recursive orchestration is real, not aspirational
• coordination primitives are explicit
• history and active operations are separated cleanly
• root behaves more like an operator and less like a passive transcript bot
• the UI gives a live operational picture instead of just a chat feed
• repo implementation work is safer and less likely to mutate Reef root by accident

Validation

This branch was exercised with repeated local tests and private live reprovisions.

Key local coverage

• tests/authority.test.ts
• tests/lieutenant.test.ts
• tests/swarm-runtime.test.ts
• tests/usage.test.ts
• tests/store.test.ts
• tests/scheduled.test.ts
• tests/logs-search.test.ts
• tests/probe.test.ts
• tests/vm-tree-history.test.ts
• src/reef.test.ts
• tests/github.test.ts
• services/services/services.test.ts

Key live validations

Live runs confirmed:

• nested lineage is preserved and no longer flattened to root
• usage rollups match the recorded tree
• sibling peer signaling works for agent siblings and swarm siblings
• store discovery and barrier waits work in real coordination flows
• scheduled checks are usable and cleaner than reminders
• root turn completion is bounded and no longer stays stuck in running
• lieutenant remains durable after done
• active/history views behave correctly in practice
• protected resource_vm behavior matches policy
• the logs browser works live
• the reef world state section is populated and useful
• repo implementation defaults outside Reef root in successful benchmark runs
• resource VM + lieutenant maintenance patterns work in practice

Follow-ups / non-blocking issues

• root still sometimes over-implements before delegating on complex repo builds
• deployment ownership reclaim should become more explicit when root takes over from a blocked child
• service/runtime operator surfaces will benefit from a dedicated Services tab later

Cross-repo context

This PR is part of a coordinated stack with:

• hdresearch/vers-fleets branch reef-v2-orchestration
• hdresearch/pi-vers branch remote-bg-process-hardening

Those PRs should be reviewed alongside this one because they provide the provisioning and runtime substrate Reef depends on.