Skip to content
Open
24 changes: 24 additions & 0 deletions docs/cli/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,30 @@ Each run produces artifacts in the output directory:
| `total_cost_usd` | Total inference cost in USD, as reported by the runtime (raw floating-point aggregate across all iterations; no fullsend-side pricing-table fallback). See [Cost data contract](../guides/infrastructure/distributed-tracing.md#cost-data-contract) |
| `num_turns` | Number of conversation turns |
| `iterations` | Number of retry iterations |
| `per_model_usage` | Per-model-spec breakdown, present only when a runtime reports one (today: `pi` with the `Agent` tool enabled). See below |

#### Per-model usage

A map from pi model spec (`anthropic-vertex/claude-opus-4-6`) to
`{requests, input_tokens, output_tokens, cache_creation_input_tokens, cache_read_input_tokens, cost_usd}`.
It exists because a pi sub-agent is a separate `pi` process whose tokens never appear in the
parent's stream, so without it `total_cost_usd` would grow with no way to attribute it.

- **What folds.** Tokens and cost, from both the parent and every child, summed across retry
iterations. Each iteration contributes one `requests` for the parent plus one per sub-agent call,
so `requests` counts inference *episodes*, not HTTP requests.
- **The invariant.** The breakdown sums to the run totals for the five fields an entry has:
`sum(cost_usd) == total_cost_usd`, and likewise for `input_tokens`, `output_tokens`,
`cache_creation_input_tokens` and `cache_read_input_tokens`. `reasoning_tokens` is a run-level
total with no per-model counterpart, so it is outside the invariant. The parent's entry is
recorded on every iteration of an `Agent`-enabled run, including ones that dispatched no
sub-agent, which is what keeps the invariant true across a retry.
- **What stays parent-only.** `num_turns` and `tool_calls` are read from the parent's stream and are
not broken down or added to per model — a child's turns and tool calls are recorded in its own
session transcript (`transcripts/<agent>-sub<seq>-*.jsonl`) instead.
- A model spec of `unknown` is a usage record that carries no model spec at all; its cost is
bucketed there rather than dropped. A dispatch rejected *before* a model was resolved writes
no record, so it never reaches the breakdown.

## OpenAI credentials on pi

Expand Down
127 changes: 115 additions & 12 deletions docs/contributing/runtime-implementation.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/guides/getting-started/choosing-a-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ sidebar_label: Choose a Runtime

# Choose an agent runtime

> **Claude Code is the stable default.** The fleet agents have run on Claude Code in production for a long time; it is what a new installation gets unless you ask for something else. **pi is in its enablement (experimental) phase** — it works end to end for `triage`, `prioritize`, `code` and `fix`, has no sub-agent tool yet (`review`/`retro` run in a single context), and its fleet pilot is still in progress. Unless you are taking part in that pilot, keep the default.
> **Claude Code is the stable default.** The fleet agents have run on Claude Code in production for a long time; it is what a new installation gets unless you ask for something else. **pi is in its enablement (experimental) phase** — it works end to end for `triage`, `prioritize`, `code` and `fix`, and `review`/`retro` now dispatch their real sub-agent roster through a fullsend-supplied `Agent`/`Task` tool, but its fleet pilot is still in progress. Unless you are taking part in that pilot, keep the default.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the specific phrase:

Suggested change
> **Claude Code is the stable default.** The fleet agents have run on Claude Code in production for a long time; it is what a new installation gets unless you ask for something else. **pi is in its enablement (experimental) phase** — it works end to end for `triage`, `prioritize`, `code` and `fix`, and `review`/`retro` now dispatch their real sub-agent roster through a fullsend-supplied `Agent`/`Task` tool, but its fleet pilot is still in progress. Unless you are taking part in that pilot, keep the default.
> **Claude Code is the stable default.** The fleet agents have run on Claude Code in production for a long time; it is what a new installation gets unless you ask for something else. **pi is in its enablement (experimental) phase** — it works end to end for `triage`, `prioritize`, `code` and `fix`, and `review`/`retro`. Unless you are taking part in the `pi` pilot, keep the default.


This page explains what the choice means and where it is made. **You do not select anything on this page** — the selection happens in the next step, [Configuring GitHub](configuring-github.md), when `fullsend github setup` prompts for the runtime (press Enter for `claude`) or when you pass `--runtime`.

Expand All @@ -15,7 +15,7 @@ Fullsend supports multiple agent runtimes. A runtime is the program that runs in
| Runtime | Status | Description | When to use |
|---------|--------|-------------|-------------|
| `claude` | **Stable (default)** | Claude Code on Vertex AI | Every production deployment — mature, full sub-agent support for `review`/`retro` |
| `pi` | Experimental (enablement phase) | [Pi](https://github.com/earendil-works/pi) — Claude on Vertex by default; any provider pi supports by model name (e.g. Gemini on Vertex with the same credentials) | Opt-in pilots only; no sub-agent tool yet, so `review`/`retro` run single-context; see [Runtimes](../../runtimes.md) for known constraints |
| `pi` | Experimental (enablement phase) | [Pi](https://github.com/earendil-works/pi) — Claude on Vertex by default; any provider pi supports by model name (e.g. Gemini on Vertex with the same credentials) | Opt-in pilots only; `Agent`/`Task` sub-agents come from a fullsend extension (children are `pi` processes) rather than from pi itself; see [Runtimes](../../runtimes.md) for known constraints |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too much detail somewhere it does not belong, probably you are already documenting this somewhere else. Remove this detail.


## When and how the runtime is selected

Expand Down
10 changes: 5 additions & 5 deletions docs/runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sandbox, the credentials, and the verdict.
| Runtime | Use it for | Status |
|---|---|---|
| **[`claude`](runtimes/claude.md)** | Production agent runs (Claude Code) | Default |
| **[`pi`](runtimes/pi.md)** | Second runtime, opt-in per repo — Claude, Grok and Gemini on Vertex; GPT via OpenAI WIF (wired, not yet exercised live) | Supported for `triage`, `prioritize`, `code`, `fix` |
| **[`pi`](runtimes/pi.md)** | Second runtime, opt-in per repo — Claude, Grok and Gemini on Vertex; GPT via OpenAI WIF (wired, not yet exercised live) | Supported for all roles |
| `dummy` | Behaviour tests — scripted ops, no inference | Internal |
| `opencode` | Not yet functional | Stub |

Expand Down Expand Up @@ -50,15 +50,15 @@ sequenceDiagram
| | Claude Code | pi |
|---|---|---|
| Models | Anthropic on Vertex | Claude, **Grok** and **Gemini** on Vertex; **GPT** via OpenAI WIF (opt-in, [not yet exercised live](runtimes/pi.md#models-and-providers)) |
| Sub-agents | Native (`Agent` tool) | Not wired — agents execute sub-agent definitions inline ([#6527](https://github.com/fullsend-ai/fullsend/issues/6527)) |
| Sub-agents | Native (`Agent` tool) | `Agent`/`Task` via a fullsend extension — children are `pi` processes with the same hooks, providers and tool allowlist ([pi runtime § Sub-agents](runtimes/pi.md#sub-agents)) |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Details where they are not needed.

Suggested change
| Sub-agents | Native (`Agent` tool) | `Agent`/`Task` via a fullsend extension — children are `pi` processes with the same hooks, providers and tool allowlist ([pi runtime § Sub-agents](runtimes/pi.md#sub-agents)) |
| Sub-agents | Native (`Agent` tool) | `Agent`/`Task` via a fullsend extension |

| Fallback model chain | `FULLSEND_FALLBACK_MODELS`, tried in order | Ignored with a warning |
| Roles | All | `review`/`retro` stay on Claude Code — they rely on sub-agent rosters |
| Roles | All | All; `review`/`retro` run their real sub-agent roster, at `--thinking medium` by default |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| Roles | All | All; `review`/`retro` run their real sub-agent roster, at `--thinking medium` by default |
| Roles | All | All; `review`/`retro` at `--thinking medium` by default |

| Effort | `--effort low..max` | `--thinking`, same levels (`high` when unset) |
| Security controls | Full matrix | Full matrix; stricter on failed-call sanitizing |

Both run unattended in the same sandbox, on the same WIF credentials, behind the same egress
allowlist. Choose `pi` when you want a non-Anthropic model; stay on `claude` when you need
sub-agents or a fallback chain.
allowlist. Choose `pi` when you want a non-Anthropic model; stay on `claude` when you need a
fallback chain.

## Selecting a runtime and model

Expand Down
5 changes: 2 additions & 3 deletions docs/runtimes/claude.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ unsupported and ignores it.

| | |
|---|---|
| Roles | All, including `review` and `retro` — they need sub-agents |
| Roles | All, including `review` and `retro` — they need sub-agents (pi covers these too, through a fullsend extension: [pi § Sub-agents](pi.md#sub-agents)) |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, too much detail where it does not belong.

| Credentials | WIF `external_account` + a refreshed OIDC token; `ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_BASE_URL` and `ANTHROPIC_VERTEX_BASE_URL` are unset so a stray key cannot redirect traffic |
| Unattended | `--dangerously-skip-permissions`; hooks wired from the harness, never from agent-writable files |
| Artifacts | `output.jsonl`, transcripts, `metrics.json` with `runtime: claude`, and `claude-debug.log` with `--debug` |
Expand All @@ -41,8 +41,7 @@ These are the places Claude Code differs from pi — useful when comparing a run
- **The agent definition *replaces* the system prompt.** `--agent` makes the agent `.md` body the
system prompt outright. pi appends it to its own default instead, so an agent that relies on
Claude Code's exact framing can read differently there.
- **Native sub-agents** via the `Agent` tool, which is why `review` and `retro` are Claude-only
today.
- **Native sub-agents** via the `Agent` tool. This is no longer Claude-only: pi serves the same `Agent`/`Task` contract from a runner-owned extension that runs each child as its own `pi` process ([pi § Sub-agents](pi.md#sub-agents)). What stays Claude-specific is that the sub-agents are *native* — no child process, no separate session dir, no per-child provider hygiene.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, comenting pi details on the claude page.

Suggested change
- **Native sub-agents** via the `Agent` tool. This is no longer Claude-only: pi serves the same `Agent`/`Task` contract from a runner-owned extension that runs each child as its own `pi` process ([pi § Sub-agents](pi.md#sub-agents)). What stays Claude-specific is that the sub-agents are *native* — no child process, no separate session dir, no per-child provider hygiene.
- **Native sub-agents** via the `Agent` tool.

- **A `CLAUDE.md` bridge is injected** when the repo has `AGENTS.md` but no `CLAUDE.md`, because
Claude Code auto-loads only the former. pi reads `AGENTS.md` natively and needs no bridge.
- **`tools:` is enforced unreliably** (≥ 2.1.119); pi enforces its `--tools` allowlist strictly. In
Expand Down
131 changes: 117 additions & 14 deletions docs/runtimes/pi.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Pi

[pi](https://github.com/earendil-works/pi) is fullsend's second agent runtime, opt-in per org or
repo. It reaches models Claude Code cannot — **Grok** and **Gemini** alongside Claude — through the
[pi](https://github.com/earendil-works/pi) is fullsend's second agent runtime, opt-in per repo. It reaches models Claude Code cannot — **Grok** and **Gemini** alongside Claude — through the
same sandbox, credentials and egress policy.

```bash
Expand Down Expand Up @@ -85,10 +84,11 @@ endpoints answer `FAILED_PRECONDITION` — so region variables are deliberately
|---|---|
| Credentials | Same WIF `external_account` + refreshed OIDC token as Claude Code for Vertex providers. `ANTHROPIC_*` unset on the Claude provider, `XAI_API_KEY` unset on the Grok one; `OPENAI_BASE_URL`/`AZURE_OPENAI_API_KEY` unset on the OpenAI one. OpenAI uses a runner-exchanged WIF token (ADR 0092) |
| Unattended | No approval prompts, stdin closed, bounded retries; a missing credential exits 1 |
| Artifacts | `output.jsonl`, `transcripts/<agent>-<ts>_<id>.jsonl`, `metrics.json` with `runtime: pi`, plus `pi-debug.log` with `--debug` |
| Extra knobs | `FULLSEND_PI_PROVIDER` (prefix for bare ids), `FULLSEND_PI_BASH_ALLOWLIST=enforce` |
| Artifacts | `output.jsonl`, `transcripts/<agent>-<ts>_<id>.jsonl` (plus `<agent>-sub<n>-…` per sub-agent and `<agent>-subagents-usage.jsonl`), `metrics.json` with `runtime: pi`, plus `pi-debug.log` with `--debug` |
| Extra knobs | `FULLSEND_PI_PROVIDER` (prefix for bare ids), `FULLSEND_PI_BASH_ALLOWLIST=enforce`, `FULLSEND_PI_SUBAGENT_THINKING` |
| Extensions | Harness `extensions:` directories, uploaded and loaded with `-e` after a tree-hash preflight ([Extensions](#extensions)) |
| Not supported | Sub-agents, fallback chains, `plugins:`, Bedrock/Azure providers |
| Sub-agents | `Agent` (alias `Task`) via a fullsend extension: children are `pi` processes with the same hooks, providers and tool allowlist ([Sub-agents](#sub-agents)) |
| Not supported | Fallback chains, `plugins:`, Bedrock/Azure providers |

## Running it locally

Expand Down Expand Up @@ -140,8 +140,9 @@ What a local pi run needs, beyond the guide:
`podman pull ghcr.io/fullsend-ai/fullsend-sandbox:latest` fixes it.
- **Platforms** — verified end to end on macOS Apple Silicon (podman machine, Homebrew `openshell`)
and Fedora with rootless Podman; the guide's platform notes apply unchanged.
- **`review` and `retro`** complete with schema-valid results but in a single context — pi has no
sub-agent tool, so the parallel reviewer roster is not exercised (see [Not yet exercised](#not-yet-exercised)).
- **`review` and `retro`** run their real sub-agent roster through the `Agent` tool; the children
default to `--thinking medium`, which keeps the roster inside the 20-minute review budget (see
[Sub-agents](#sub-agents)).
- **Knobs** — `FULLSEND_PI_PROVIDER` sets the provider for bare model ids (default
`anthropic-vertex`); `FULLSEND_PI_BASH_ALLOWLIST=enforce` makes the Bash first-token allowlist
block instead of warn.
Expand Down Expand Up @@ -265,20 +266,122 @@ How the runner protects this path — the tree hash, the loader cache, the symli
deny-list — is in
[Runtime Implementation § Pi extensions](../contributing/runtime-implementation.md#pi-extensions-adr-0094).

## Sub-agents

The `Agent` tool (registered under its legacy alias `Task` as well) comes from a runner-owned pi
extension, `fullsend-agent.js`, so skills written for Claude Code's sub-agent roster — `pr-review`,
`retro-analysis` — dispatch unchanged. Each child is its own `pi --print` process.

| Parameter | Meaning |
|---|---|
| `prompt` (required) | The whole task. The child starts with no memory of the conversation, so the prompt must carry its own context package |
| `description` | Short label; shows in the run log |
| `model` | A model this run can serve (see below). Omitted → the parent's model |
| `subagent_type` | `Explore` gives a read-only child (`read`, `grep`, `find`, `ls`, intersected with the parent's set); anything else, or omitted, gives the parent's tool set |
| `run_in_background` | Accepted and ignored — a child always runs to completion inside the call |

The call returns the child's final assistant message, trimmed and capped at 64 KB with a
`[truncated]` marker.

### Choosing a model

For a run that reaches all three Vertex providers:

| `model` | The child runs on |
|---|---|
| `sonnet` (also `opus`, `haiku`) | Claude on Vertex, whatever provider the parent runs on |
| `claude-sonnet-4-6` | the same — a bare Claude id resolves through that alias table, and a persona-style `@default` suffix is dropped |
| `google-vertex/gemini-3.7-flash` | Gemini, on pi's built-in provider |
| `xai/grok-4.6` | Grok on Vertex — normalized to `xai-vertex/xai/grok-4.6`, as the runner does for the parent |

Anything else is **rejected, with the accepted forms listed in the error**, so the orchestrator can
correct itself instead of losing the dispatch. The accepted set is closed — the run's model table,
the parent's own spec, and the ids registered for a provider that has no table entry — rather than a
provider-prefix check, so an id the model invented (`google-vertex/gemini-9`,
`anthropic-vertex/claude-sonnet-4-20250514`) is refused even under a provider the run can reach. A
trailing `:<thinking level>` is dropped rather than passed through.

### Running children in parallel

Put several `Agent` calls in one assistant message: pi runs sibling tool calls from one message
concurrently. At most four children run at once and the rest queue. The runtime note appended to the
agent's system prompt says so, so a skill that asks for "dispatch these in parallel" gets it.

### What a child inherits, and what it does not

A child starts with the parent's posture — `--no-approve`, `--no-extensions` with an explicit `-e`
list, no prompt templates or themes, its own session dir, and a `--tools` allowlist
(`--no-builtin-tools` when that allowlist is empty). It inherits:

- **The sandbox hooks.** Its `-e` list carries the vendored provider extensions and the hook adapter,
so PreToolUse/PostToolUse hooks and the Bash allowlist apply inside sub-agents too — as they do on
Claude Code, where the same hooks run on `Agent` calls.
- **The parent's tool set**, minus `Agent`/`Task`: a child cannot dispatch children of its own.

It does not inherit:

- **Harness `extensions:`.** A child's `-e` list is fixed at bootstrap, so a tool one of your
declared extensions registers is not available inside a sub-agent.
- **The parent's system prompt.** Children get a short sub-agent role note instead of the
orchestrator persona, whose "make several `Agent` calls in one message" advice a child cannot act
on.
- **Provider credentials it does not use.** The environment is rebuilt for the provider the child
resolved to, so a Claude child under a Grok parent carries no stray `ANTHROPIC_API_KEY`.

### Thinking level

Children run at `--thinking medium`, not the parent's `high`: a full `pr-review` roster at `high`
overran the 20-minute review budget. Override with
`FULLSEND_PI_SUBAGENT_THINKING=<off|minimal|low|medium|high|xhigh|max>`; an unrecognised value warns
and falls back to `medium`.

### Where the output lands

- **Transcripts** — `transcripts/<agent>-sub<seq>-<basename>.jsonl`, one per child. The sequence
number is the call's, so children sharing a session basename do not collide.
- **Usage** — one JSON line per child (model, usage, stop reason, duration) in
`transcripts/<agent>-subagents-usage.jsonl`.
- **`metrics.json`** — the totals include the children, and `per_model_usage` attributes them per
model spec, with the parent's own iteration as one entry so the breakdown sums to the totals
([`fullsend run` § metrics.json](../cli/run.md#per-model-usage)). A record with no model spec is
bucketed under `unknown`.
- **Run log** — `[fullsend-agent] #<seq> <model> start "<description>"` and
`[fullsend-agent] #<seq> done <ms>ms <stopReason>` per child.

### Turning it off

The tool is enabled when the agent's definition has no `tools:` frontmatter (the default set, as
under Claude Code) or lists `Agent`/`Task`. An agent that lists tools without them gets no `Agent`
tool, and the runtime note telling it to execute sub-agent definitions itself, in order.

### Troubleshooting sub-agents

| Symptom | Cause | Fix |
|---|---|---|
| `model "<spec>": ...; use opus, sonnet, haiku, or one of ...` | The `model` argument is not one this run can serve | Use one of the forms the message lists, or omit `model` to inherit the parent's |
| `manifest changed since load; refusing to dispatch` | `fullsend-manifest.json` changed after the extension read it | Runner-owned config was rewritten inside the sandbox — treat it as tampering, not a transient |
| `hook adapter changed since load; refusing to dispatch` | `fullsend-hooks.js` changed after bootstrap recorded its digest | The same: the child would otherwise have come up unhooked |
| A child call fails after 15 minutes | The per-child deadline; the child is signalled and reaped | Narrow the child's prompt, or split the task across more children |
| A child call reports `error` or `aborted` | The child's own run failed — model error, non-zero exit, or no `agent_end` | Read that child's transcript under `transcripts/<agent>-sub<seq>-*.jsonl` |

How children are launched and kept honest — prompt delivery, the stop sequence, the per-dispatch
digest re-checks — is in
[Runtime Implementation § Pi sub-agents](../contributing/runtime-implementation.md#pi-sub-agents-the-agent-tool-contract).

## Not yet exercised

`runtime: pi` is selectable and has been run end to end, but no **fleet lifecycle** run on Vertex is
recorded yet. Pilot on a disposable repo with `triage`/`prioritize` before `code`/`fix`. `review` and
`retro` run to schema-valid results, but in a **single context**: pi has no sub-agent tool, so the
parallel persona roster and its per-persona models are never exercised — treat them as unsupported
for that purpose. `extension_error` events are not mapped.
recorded yet. Pilot on a disposable repo with `triage`/`prioritize` before `code`/`fix`. The
sub-agent roster of `review`/`retro` has been exercised locally, not yet on a fleet lifecycle run —
watch the wall clock on the first one (see [Sub-agents](#sub-agents)). `extension_error` events are
not mapped.

## Troubleshooting

**The model is not found, or the provider is missing.** A pi provider comes from an extension loaded
with `-e`, so an extension that did not load takes its provider with it. The table in
[Extensions § Troubleshooting extensions](#troubleshooting-extensions) separates the two ways that happens — the loud
one (`Failed to load extension`, exit 1) and the silent one (pi exits 0 having loaded nothing).
with `-e`, so an extension that did not load takes its provider with it. The table under
[Extensions](#troubleshooting-extensions) separates the two ways that happens — the loud one
(`Failed to load extension`, exit 1) and the silent one (pi exits 0 having loaded nothing).

**`No API key found for <provider>`.** The provider is registered but its credentials did not
resolve. For Vertex providers that means ADC — check the project variable for *that* provider in the
Expand Down
Loading
Loading