feat(sdk): upgrade to claude-agent-sdk 0.2.150 and adopt what it changed

CHANGELOG.md

@@ -7,6 +7,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- **Upgraded `claude-agent-sdk` to `>=0.2.150,<0.3`** (from `0.2.137`). Thirteen releases; twelve of them are bundled-CLI bumps carrying Claude Code **2.1.229 → 2.1.257**. The one substantive release is 0.2.140, and three of its four features land here (`ResultError`, `forward_subagent_text`, and the MCP 2.x widening that forced the pin below); `can_use_tool` for string prompts is not used — claudetm runs `permission_mode="bypassPermissions"` and has no permission callback to install.
+- **`CLAUDETM_HIVE_MAX_PARALLEL` default 10 → 6.** This constant is **prompt-visible**: it is interpolated verbatim into the fan-out brief as the ceiling a lead may dispatch up to, so it is not an internal guard that can be tuned quietly. Ten invited a split wider than a single shared checkout on one box rewards — every worker is a full agent process re-reading the repo on the same cores, and past a handful the cold starts and the lead's own verification pass cost more than the concurrency returns. Still a ceiling and never a target; zero remains the right answer for most tasks.
+- **The brief now states that the team's *composition* is the lead's, and that nothing rewards variety.** Up to the ceiling, workers may be N of the same kind, one each of N different kinds, or any mix — four concurrent `hive-worker`s is exactly as legitimate as a `backend-dev`, a `frontend-dev` and two `hive-worker`s. What selects a worker is the piece it is handed: the specialist whose description fits it, else generic `hive-worker`. Two pieces suiting the same specialist get two instances of it. Previously the brief named the ceiling and the specialist-first rule but never said whether repeating a type was allowed, which reads as if it is not.
+- **Regenerated `requirements.txt`**, a committed `uv pip compile` artifact that had gone stale at `claude-agent-sdk==0.1.35` — roughly 115 releases behind, and unusable against this codebase. Nothing in CI or the Dockerfile consumes it, but claudetm's own repo-setup handler runs `uv pip install -r requirements.txt` whenever it finds one, so the file was a live trap for anyone it was pointed at.
+
+### Added
+
+- **A fanned-out session now shows its team, not just its lines** (`core/hive_roster.py`, `CLAUDETM_HIVE_ROSTER_INTERVAL_SEC`, default 60s, `0` disables). The per-worker `↳ [hive-worker#2]` prefixes answer "who said this" and cannot answer the questions you actually have while watching a hive: how many workers are live, what each is on right now, how long each has been going, and what each has burned. The roster tracks every worker from its dispatch block to its returning tool result and renders a compact block — forced when the team's shape changes, throttled otherwise:
+
+  ```
+  Hive: 2 workers active, 1 done
+    ~ hive-worker#1   Bash uv run pytest tests/core/test_auth.py   5m 11s   in 410.2k out 8.1k
+    ~ backend-dev#2   Edit src/api/routes.py                       5m 11s   in 180.2k out 6.0k
+    + hive-worker#3   done                                         5m 11s   in  41.0k out 2.2k
+  ```
+
+  Two stream facts, both established by probing a live session rather than by reading, decide how this is fed. **A dispatch's tool result is an acknowledgement, not a completion**: the `ToolResultBlock` for an `Agent` call arrives ~0.1s after the dispatch while that worker's own messages keep arriving for the next 40 seconds — the first cut read it as "returned" and rendered `3 done` at `0s` elapsed with all three still working. Only a *failed* dispatch now retires a worker. And the roster **adopts an unfamiliar tool-use id** by design (a worker can speak before its spawning block is processed), so completion is additionally gated on an id already recorded as a dispatch — otherwise every failed ordinary Read/Bash call invents a phantom worker on sessions with no hive at all.
+
+  Per-worker tokens come from `AssistantMessage.usage` on messages carrying `parent_tool_use_id` — the only per-subagent figure the stream offers, since the terminal `ResultMessage` aggregates the lead and every worker into the one total that cannot say *which* worker spent it. The SDK does not document that field's exact meaning, so it is best-effort: malformed values are ignored rather than reported as a number.
+
+- **The hive ceiling is enforced by the runtime, not just stated in the prompt.** `CLAUDETM_HIVE_MAX_PARALLEL` had always been prose — a number interpolated into the brief with nothing behind it — and this codebase has already measured what a prose-only rule is worth (leads ignored the "never background a worker" instruction in 27% of dispatches, which is why that one is pinned on the agent definition). The bundled CLI enforces a cap of its own: it hands out concurrency slots and refuses the overflow dispatch with `Concurrent subagent limit reached`, reading `CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS` and defaulting to 20 when unset. claudetm now passes its own ceiling there, so the brief and the runtime agree on one number.
+
+- **Workers' own prose reaches the session log, not just their tool calls** (`forward_subagent_text`, new in SDK 0.2.140; `CLAUDETM_FORWARD_SUBAGENT_TEXT`, default on). The SDK forwards a subagent's tool calls unasked but withheld its text and thinking, which left a fanned-out session half-visible: you could see *that* a worker edited a file and never what it was trying to do — on a run where the worker's final message is the only thing that comes back to the lead, and the log is the only forensic record. The rendering it needs already existed: stable per-worker colour and `#n` ordinal, and subagent text displayed but never accumulated into the lead's own result. Forwarding is display-only and costs no tokens — those blocks were generated either way; the only price is log volume. Applied solely to sessions that may fan out, since a session denied the dispatch tools has no subagent to forward.
+
+### Fixed
+
+- **`mcp` is now pinned `<2`, because upgrading the SDK breaks `claudetm-mcp` without it.** mcp 2.x renamed `FastMCP` to `MCPServer` and removed `mcp.server.fastmcp`, which every module under `claude_task_master/mcp/` imports. claudetm's extra was the unbounded `mcp>=1.26.0`; until 0.2.140 the SDK's own `mcp<2.0.0` pin was silently holding the line for us, and 0.2.140 widened it to `mcp<3.0.0`. Verified: `pip install "claude-task-master[mcp]==0.1.89"` resolves **mcp 2.1.1** today, and `from mcp.server.fastmcp import FastMCP` then raises `ModuleNotFoundError`. Lift the pin only together with the MCPServer migration.
+- **Error classification read the prose and could not see the payload** (`core/agent_error_classify.py`, extracted from `agent_query_helpers`). The verdict decides whether an unattended run **retries or dies** — `TRANSIENT_ERRORS` members retry under the failure budget, everything else propagates — and it was derived from substring matching on `str(error)` alone. SDK 0.2.140 raises `ResultError` carrying `api_error_status`, `subtype`, `terminal_reason`, `errors` and `result`, so the structured payload is now consulted first: 429 → rate limit, 401/403 → auth, 404 *with* a model mentioned → the fallback chain, 408/504 → timeout, any other 5xx → server error. An HTTP status is a fact; a substring is a guess.
+
+  `str(ResultError)` is **only** `"Claude Code returned an error result: <subtype> (exit code: 1)"` — the prose naming the actual failure lives in `errors`/`result`. Folding the payload into the searched text is what makes **529 overloaded** a retryable `APIServerError` (it appears nowhere in `str()`, so the old rules classified it as an unrecognised error) and what lets `Connection closed mid-response` — the blip that once ended a 22-task unattended run at task 1 — classify as `APIConnectionError` at all.
+
+- **A bare `401`/`403` in echoed command output is no longer an auth error.** Folding the `result`/`errors` payload into the searched text is what lets a real failure be recognised, and it drags tool and command output in with it — so `wrote 403 bytes` or `exit 401` matched a digit-boundary rule. Auth status matching now requires a status-like word nearby; the 5xx rule deliberately does not need to, because a false 5xx is merely retried while a false auth error is not retryable and ends the run. `Forbidden` is also matched as a word, so `403 Forbidden` survives without the digits.
+- **Two loose substring rules, each wrong in one direction.** `"500" in text` also matched `request took 1500ms`, promoting a latency message to a retryable `APIServerError`; `"auth" in text` also matched **`Co-Authored-By`**, which appears in every commit message this project writes, so any error echoing a git command became a *non*-retryable `APIAuthenticationError` that ended the run. Status codes and auth phrasings now match on token boundaries, pinned by named regression tests.
+
 ## [0.1.89] - 2026-08-14
 
 ### Fixed

CLAUDE.md

@@ -233,6 +233,30 @@ Unfinished → the task is **not** checked off and the same task re-runs, with a
 
 Relatedly, an unclassified `QueryExecutionError` (a CLI crash whose text carries no keyword `_classify_api_error` recognises) is no longer fatal on sight: it retries under the same failure budget as a connection error (`rate_limit_config.max_retries`), and only a persistent one raises `ConsecutiveFailuresError`.
 
+### Error classification reads the payload, not the prose (`core/agent_error_classify.py`)
+
+The verdict decides whether an unattended run **retries or dies** — `TRANSIENT_ERRORS` members are
+retried under the failure budget, everything else propagates — so it is no longer derived from
+substring matching alone. `claude-agent-sdk` >= 0.2.140 raises **`ResultError`** (a `ProcessError`
+subclass) when the CLI exits after a terminal error result, carrying `api_error_status`, `subtype`,
+`terminal_reason`, `errors` and `result`. An HTTP status is a fact; a substring is a guess, so the
+structured payload is consulted first: 429 → rate limit, 401/403 → auth, 404 *with* a model
+mentioned → the fallback chain, 408/504 → timeout, any other 5xx → server error.
+
+`str(ResultError)` is **only** `"Claude Code returned an error result: <subtype> (exit code: 1)"` —
+the prose naming the actual failure lives in `errors`/`result`. So the payload is folded into the
+searched text as well, which is what lets `Connection closed mid-response` (the blip documented
+above as ending a 22-task run) classify as a retryable `APIConnectionError` at all, and what makes
+**529 overloaded** — invisible to the old rules, absent from `str()` — a retryable `APIServerError`.
+
+The text rules now match on **token boundaries**. Two one-way misclassifications lived in the loose
+version: `"500" in text` also matched `request took 1500ms`, promoting a latency message to a
+retryable server error; and `"auth" in text` also matched **`Co-Authored-By`**, which appears in
+every commit message this project writes, so any error echoing a git command became a
+*non*-retryable `APIAuthenticationError` that ended the run. Both are pinned by named regression
+tests. The module reads the SDK fields by `getattr` and never imports `claude_agent_sdk` — it must
+stay importable in exactly the situation some of these errors describe.
+
 ### Accumulated context is capped where it is read
 
 `context.md` grows by a summary per session and never shrinks, so what enters a prompt has to be
@@ -345,7 +369,11 @@ A triplicated `plan.md` still runs its work N times; the parser only guarantees
 - **The seam is the API, not the path.** Disjoint file sets are necessary and not sufficient: a piece that renames an export, changes a signature or edits a shared type breaks every file importing it, *including files no worker owns*. The lead lists, per piece, what it changes that something else reads, and either pulls every caller into that set or keeps the piece itself. (Observed live: one worker renamed a field across the files it owned, an unowned file kept the old name, and all six workers reported success.)
 - **Whether workers may run checks at all is the lead's call, stated in each brief.** Concurrent scoped test/lint runs are a property of *the project* — a shared database or fixture schema, a fixed port, one build or coverage directory, a lock — so the lead looks, decides, and writes the exact scoped command into each brief or explicitly "none". Either way it runs the full gate once at the end. Commands that *write* count as writing: a repo-wide formatter/autofixer rewrites every worker's files at once, and an emitting build or codegen step has workers racing over shared artifacts — both are the lead's alone, never run while a worker is live.
 - **Fan-out is not free, and the brief says so.** Every worker pays a full cold start re-reading the repo, and its final message is the only thing that comes back. A sibling project was observed spawning four workers for four one-line edits: four cold starts for work one agent finishes in a single pass. That is the failure the brief argues against, at length, because it is the only lever there is.
-- **`CLAUDETM_HIVE_MAX_PARALLEL` (default 10) is a safety ceiling, not a target** (`core/hive.py`). It bounds concurrent workers — one lead plus up to N — and nothing else. A typo in the env var never ends a run: anything unset, unparseable or `<= 0` falls back to the default.
+- **`CLAUDETM_HIVE_MAX_PARALLEL` (default 6) is a safety ceiling, not a target** (`core/hive.py`). It bounds concurrent workers — one lead plus up to N — and nothing else. A typo in the env var never ends a run: anything unset, unparseable or `<= 0` falls back to the default. The number is **prompt-visible**: it is interpolated verbatim into the fan-out brief, so changing the constant changes what every lead is told it may dispatch — which is why the default moved from 10 to 6, a width a single shared checkout on one box actually rewards.
+- **The ceiling is now enforced, not merely stated.** It had always been prose — a number interpolated into the brief with nothing behind it — and this repo has already measured what a prose-only rule is worth (leads ignored "never background a worker" in 27% of dispatches, which is why that one is pinned on the definition). The bundled CLI enforces its own cap: it hands out concurrency slots and refuses the overflow with `Concurrent subagent limit reached`, reading `CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS` and defaulting to **20** when unset (verified in the bundled binary: `var _e=20; return env.CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS ?? _e`). `_execute_query` now passes `hive_max_parallel()` into the CLI env, so the brief and the runtime finally agree on one number. Related and deliberately *not* set: `CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH` (default 3) would structurally stop a *project specialist* — which claudetm no longer passes definitions for, so it carries no `disallowedTools` — from spawning its own workers, but whether depth 1 also forbids the lead's own dispatch was not verified, and shipping an unverified value that could disable fan-out outright is the worse risk.
+- **A fanned-out session shows its team, not just its lines** (`core/hive_roster.py`). Per-worker prefixes answer "who said this"; they cannot answer "how many workers are live, what is each on, and what has each burned" — the questions you actually have while watching a hive. `HiveRoster` tracks each worker from its dispatch block to its returning tool result and renders a compact block: `name#n`, current activity, elapsed, tokens. It is pure state plus a renderer — it prints nothing and imports neither `console` nor the SDK — and `MessageProcessor` is the only thing that turns it into output, forced when the team's shape changes (a dispatch, a return) and otherwise throttled by `CLAUDETM_HIVE_ROSTER_INTERVAL_SEC` (60s; `0` disables). Per-worker tokens come from `AssistantMessage.usage` on messages carrying `parent_tool_use_id` — the only per-subagent figure the stream offers, since the terminal `ResultMessage` aggregates the lead and every worker into the one total that cannot say *which* worker spent it. The SDK does not document that field's exact meaning, so it is best-effort: anything malformed is ignored rather than reported as a number. **A dispatch's tool result is an acknowledgement, not a completion** — measured against a live session, the `ToolResultBlock` for an `Agent` call arrives ~0.1s after the dispatch while that worker's own messages keep arriving for the next 40 seconds. Reading it as "the worker returned" is what produced a roster that jumped straight to `3 done` at `0s` elapsed with all three still working. So only a **failed** dispatch retires a worker (a refused spawn — the CLI's `Concurrent subagent limit reached` — or a crash): that worker never ran, and it is the one completion the block stream states outright. Nothing marks when a *successful* worker stopped, and inventing that signal was the original bug. The roster also deliberately **adopts an unfamiliar tool-use id** (a worker can speak before its spawning block is processed), which makes the call sites load-bearing — handing it every errored top-level `ToolResultBlock` would invent a phantom row per failed Read/Bash call on sessions with no hive at all, so it is gated on an id already recorded as a dispatch. Both are pinned by named regression tests.
+- **Within the ceiling, the composition is the lead's and nothing rewards variety.** Up to `max_parallel` workers at once, and they may be N of the same kind, one each of N different kinds, or any mix — four concurrent `hive-worker`s is exactly as legitimate as a `backend-dev`, a `frontend-dev` and two `hive-worker`s. What selects each worker is the piece it is handed: the specialist whose description fits it, else generic `hive-worker`. Two pieces suiting the same specialist get two instances of it.
+- **Workers' own prose reaches the log, not just their tool calls** (`CLAUDETM_FORWARD_SUBAGENT_TEXT`, default on). The SDK forwards a subagent's tool calls unasked but withholds its text and thinking behind `forward_subagent_text` (claude-agent-sdk >= 0.2.140), which left a fanned-out session half-visible: you could see that a worker edited a file and never what it was trying to do. The rendering it needs already existed — stable per-worker colour and `#n` ordinal, and subagent text displayed but never accumulated into the lead's result. Forwarding is display-only and costs no tokens; the only price is log volume.
 - **One checkout, shared by everyone.** Workers work directly in this same tree. Never a git worktree, never a clone, never a per-agent copy — the work has to land in the tree the lead commits from. The exclusive file sets in each worker's brief are the only lock there is; a worker that needs a file it does not own STOPS and reports the collision rather than resolving it silently.
 - **The lead alone runs git.** Workers read, edit and run narrow checks; they never `add`, `commit`, `branch`, `checkout`, `stash` or `push`. That is what makes the shared checkout safe — one writer to the index, so no `.git/index.lock` contention and no half-staged tree — and it keeps history a single authored series. The lead waits for every worker to return, verifies on disk (`git status`, read the changed files), runs the **full** project gate once, and only then commits.
 - **A worker's report is not evidence.** The lead never saw its tool calls. A worker that narrated a change without writing it is a real failure mode, so the lead re-checks on disk and re-does or re-assigns that piece itself.