ATV-Phoenix

Copilot proposes the code. Phoenix decides whether the evidence is good enough.

Phoenix is a verification and recovery harness for GitHub Copilot and Microsoft Scout. It runs real checks, recovers from failures, and records proof that the work went from failing to passing.

Journey | Install | Quick start | Features | Self-checks | Evidence | Docs

Phoenix replaces "the agent says it is done" with an external control loop:

1. Define a runnable acceptance check.
2. Observe the check fail.
3. Let the agent edit and recover.
4. Re-run the same check.
5. Ship only when the hash-chained trace shows the check was red before the fix, turned green afterward, and is still green on the final recheck.

Use Phoenix for bug fixes, refactors, PR work, and unattended jobs where a runnable check can define the outcome.

Intent engineering across three loops

Phoenix converts a raw request into an Intent Contract before implementation. phoenix-goal formalizes one goal; phoenix-intent decomposes up to five related goals. Each gets a runnable acceptance check that must be observed RED before code changes begin. phoenix-plan turns the contract into checked backlog items. Changing the check later is an explicit re-scope and requires a new RED baseline.

• Inner loop: self-healing - one change. phoenix_sense checks state, phoenix_snapshot saves only a passing baseline, and phoenix_heal rolls back or retries up to three times. Recovery counts only after an external recheck passes.
• Outer loop: goal execution - one goal. phoenix-auto selects the next lifecycle step while phoenix-ralph works the checked backlog across fresh contexts, filesystem memory, and fixed budgets. The loop stops only when phoenix_accept proves the top-level check went RED to GREEN.
• Learning loop: measured improvement - many verified runs. phoenix_learn uses public data to propose, development data to select, and a held-out private split to measure the selected candidate once. Eligibility requires at least 20 held-out examples, a 10-point gain, +2 net correct, zero regressions, and clean anti-gaming checks.

This is self-learning, not self-modifying. The gate decides eligibility; it never adopts. A human must approve any skill or prompt change, which then needs its own failure-first Phoenix proof.

The harness checks itself

A harness that verifies your code but not its own is asking for trust it refuses to give. Each rule below was added after a real defect proved it was missing.

Guarantee	What it prevents
A hang goes RED	timeout_secs sat in the public tool schema and was read nowhere, so a check declaring a 2-second bound came back green after 14 seconds. A stall was the one failure that looked like patience. Hung checks are now killed with their whole process tree and reported RED.
Killed is not the same as failed	A process can hit its timeout and exit 0 because it trapped the signal. Timeout and exit code are now separate facts, so "the OOM killer took it" no longer looks identical to "it exited -1".
The useful half of the output survives	Cutting subprocess output at a fixed byte offset crashed on non-ASCII characters — at random, and only ever while a check was already failing. Output is now cut on a character boundary and keeps the end, where the actual error is, instead of the start, where the banner is.
Every module states its rule	Each file under src/ either states a rule that can be broken, or records why it has none. 27 of 27 decided. The audit finds modules by walking the source, not by reading a list, so a new module is covered the day it lands — and a sweep that finds nothing fails instead of quietly passing.

Run the audit yourself:

cargo test --test module_invariants

The "no rule here, and here is why" answer is deliberate and expected to be common. A rule demanding a real invariant from every module turns into paperwork within a month. This one demands a decision, written down where a reviewer can argue with it.

Install

Requires Git, GitHub Copilot CLI, Python 3, and Rust.

git clone https://github.com/All-The-Vibes/ATV-Phoenix
cd ATV-Phoenix
python .copilot-plugin/skills/phoenix-setup/setup.py --repo .

The installer builds phoenix-mcp, registers the MCP server, installs the Phoenix agent and skill pack, and runs an install-integrity check. It also attempts to install TokenMasterX, the maintainer's graph-routing plugin. TokenMasterX requires graphify on PATH; setup prints the exact follow-up command when that optional dependency is missing. Restart the Copilot CLI session after setup.

If an upgrade or host change breaks the install:

# Windows
.\target\release\phoenix-mcp.exe doctor --fix

# macOS / Linux
./target/release/phoenix-mcp doctor --fix

If phoenix_sense is denied with "could not request permission from user", the host registered Phoenix but did not approve it for this folder. Run phoenix-mcp doctor --permissions --fix once to grant the per-folder approvals while preserving other entries and backing up the prior config. As a fallback, phoenix-mcp sense @check.json uses the same gate ledger without requiring MCP approval.

Quick start

Start Copilot with the Phoenix agent:

copilot --agent phoenix

This starts an interactive session. Phoenix acts on the task you give it; it does not start an unattended loop by itself. If the agent does not load, run phoenix-mcp doctor --fix, restart Copilot, and retry.

Give it a task with a concrete check:

Fix the failing test. Use `python -m pytest tests/test_widget.py -q` as the
acceptance check. Do not finish until phoenix_accept proves red to green.

Phoenix should:

• run the check and record the failing result;
• edit the smallest relevant surface;
• re-run the same check;
• recover or roll back if it stays red;
• call phoenix_accept only after the trace proves the check went red to green.

The audit trail lives in .phoenix/trace.jsonl. Verify it directly with:

# Windows
.\target\release\phoenix-mcp.exe verify-trace

# macOS / Linux
./target/release/phoenix-mcp verify-trace

The MCP completion tool is phoenix_accept. Its direct phoenix-mcp accept CLI form is:

.\target\release\phoenix-mcp.exe accept @check.json

For a full first project walkthrough, see the developer journey.

Core features

The proof stack

Capability	What it does
1. Intent engineering	phoenix-goal, phoenix-intent, and phoenix-plan convert direction into checked contracts and backlogs before implementation starts.
2. Objective checks	phoenix_sense evaluates command exits, file hashes, regexes, prompt manifests, and UI behavior without asking an LLM to grade itself.
3. Self-healing	phoenix_snapshot saves only passing state. phoenix_heal performs bounded rollback or retry, then confirms recovery with an external recheck.
4. Proven completion	phoenix_accept refuses any check never observed failing, and returns success only when an intact trace proves failure first and success now.
5. Bounded execution	A check that hangs is killed with its process tree and reported RED. A timeout and an exit code stay separate facts, so a process that traps the signal and exits 0 cannot read as a clean pass.

Beyond the core loop

Capability	What it does
Long-horizon execution	phoenix-goal formalizes the finish line, phoenix-auto chooses the next lifecycle step, and phoenix-ralph persists across fresh-context iterations.
Graph-aware context	phoenix-context asks TokenMasterX for call relationships and change impact instead of repeatedly scanning whole directories.
Portable knowledge	phoenix-okf turns code and external knowledge into Open Knowledge Format (OKF) bundles: linked Markdown that can be validated, reviewed, and reused.
Measured learning	phoenix_learn evaluates candidate prompt and skill improvements on held-out outcomes; adoption remains human-gated.
Install integrity	phoenix-mcp doctor detects drift in the agent, skills, MCP registration, and binary freshness, then repairs it with --fix.
Self-audit	Every module either states a rule that can be broken, or records why it has none — enforced by a test that walks the source rather than reading a hand-maintained list. See the harness checks itself.
Multiple hosts	GitHub Copilot uses the MCP server and agent pack. Microsoft Scout uses the same Rust binary through the CLI adapter in dist/scout.

Browse the documented lifecycle in docs/skills.md. Skill names use hyphens (phoenix-goal); MCP and CLI tool names use underscores (phoenix_sense).

The trace, not the model's success message, is the source of truth.

Autonomous workflows

• Interactive: copilot --agent phoenix works on the task and permissions you provide.
• Autonomous: phoenix-goal formalizes the outcome, phoenix-auto routes by objective state, and phoenix-ralph persists across fresh contexts and fixed budgets.

See docs/autonomous-workflows.md for state files and drivers.

Evidence

Phoenix ships its evaluation inputs and outputs in the repository. The results are directional, not universal claims.

Evaluation	Result	Scope
Pinned paired harness	Phoenix 38/45 objective passes vs control 34/45; silent failures 7/45 vs 11/45	90 real gpt-5.6-sol calls, 9 tasks, 5 seeds, paired arms, independent sealed and adversarial checks
Silent-failure experiment	Silent failures 40% to 0%, with zero regressions	20 live Copilot sessions, one older model/CLI configuration, deterministic checkers
SWE-bench-style evaluation — retired, kept for the record	Overall resolved rate 78% to 100%; underspecified tier 50% to 100%	9 constructed tasks, one repetition, explicit test gate in the Phoenix arm; not the official SWE-bench dataset. Last scored 2026-07-03 and no longer maintained as evidence — the benchmark is out of scope for this project. Read as a historical measurement, not a current claim
OKF evaluation	Index-first retrieval used 31x fewer tokens than raw graph.json	50-file bundle. A cost measurement only: the eval counts tokens and assumes each strategy retrieves enough to answer. Sufficiency is not measured. Benefit is strongest across repeated and larger-context work
Measured-learning gate	Candidates need n >= 20, +10 percentage points, +2 net correct, and zero regressions	Deterministic offline gate; it decides eligibility and never auto-adopts

The paired harness stores the exact source commit, model, runner, environment, task-set, seed, verifier, and raw-run hashes. Inspect raw-runs.jsonl for every row.

What ships

• Rust MCP and CLI spine plus a verification-gated lifecycle skill pack.
• PowerShell and Bash Ralph drivers, Copilot setup/repair, and a Microsoft Scout adapter.
• Measured-gain learning gate plus TokenMasterX graph integration from the same maintainer.
• Reproducible tests, raw evidence, and the full BUILDLOG.md.

Honest limits

• Phoenix proves the check you give it. A weak check can still prove the wrong outcome.
• A check can go red for the wrong reason. RED from a missing test file is not RED from a failing property, and the trace cannot tell the two apart. tests/test_e2e_proof_is_not_vacuous.py guards the end-to-end proof against that shape after it happened once.
• The swe-bench-style gate is retired and no longer gates anything. Phoenix resolved 9 of 9 on the constructed set against its own recorded baseline of 9 of 9, so the gate could catch a regression and could never show an improvement. Rather than re-baseline a benchmark that is out of scope for this project, it was withdrawn as evidence; the recorded result stays in evals/ as history. Tier 3 merge gating now runs under the instrument-validity rule in MISSION.md, which reports UNKNOWN for a measurement that is missing, void, over 14 days old, or saturated.
• Recovery is bounded rollback and retry, not general autonomous repair.
• Command timeouts are enforced. sense spawns the check with pipes, drains its output on separate threads so a full buffer cannot deadlock the wait, polls against the deadline, then kills the whole process tree and waits for it to actually die. A kill that returns before the work stops trades a hung check for an orphaned process, so it waits.
• The published evaluations use small constructed task sets and single models. Treat the deltas as evidence for these conditions, not as a universal ranking.
• Phoenix runs when you invoke the agent or CLI. It is not a background repository daemon.
• Self-learning is measured and human-gated. Phoenix does not rewrite or adopt its own instructions.
• Use setup.py today. The Copilot CLI marketplace/plugin-install path is scaffolded but not yet verified end to end.

Documentation

• Developer journey: build a project from intent to demonstrated outcome.
• Autonomous workflows: goal, Ralph, state, budgets, and proof.
• Skill catalog: every lifecycle and craft skill.
• Intent to outcome: architecture, research questions, and design rationale.
• Changelog: shipped changes by release.
• Build log: the full engineering record, including failures and recoveries.

License

MIT. See LICENSE.

TokenMasterX is owned by the same maintainer and included under its MIT license in vendor/token-master.