autocoder

An autonomous coding agent whose interesting part is how it stops.

$ python -m autocoder ./my-project

working on a copy at /tmp/my-project.autocoder
outcome: no_progress
attempts 2, edits 2 applied / 0 refused, 3 test runs
baseline  fail   0 passed, 1 failed
  #1 target.py                    applied  fail   no model configured
  #2 target.py                    applied  fail   no model configured
final     fail   0 passed, 1 failed

That is a successful run. The agent could not fix the bug, noticed it was making no progress after two attempts, and said so — rather than burning fifty model calls to reach the same place, or quietly editing the test until it passed.

The loop itself is the easy part: read the failure, propose an edit, run the tests, repeat. Every demo has one. What separates a demo from something you would let near a repo is the three ways that loop goes wrong, and only one of them is loud.

Part of Agent Lab · Vol 1 · Project 05 — the last of Vol 1. Standard library only. 24 tests, no API key, no network.

---

Quickstart

git clone https://github.com/dev48v/autocoder
cd autocoder
python -m venv .venv && . .venv/Scripts/activate   # Linux/macOS: . .venv/bin/activate
pip install -e ".[dev]"

python -m autocoder ./some-project     # runs on a COPY, never the original
pytest -q                              # 24 tests

---

The three failure modes

1. It cheats

The shortest path to a green test suite is editing the test.

# the cheating proposer, which is in the repo as a test fixture
Proposal("tests/test_target.py", "def test_target():\n    assert True\n")

This has to fail in code, not in a prompt. A system prompt saying "do not modify tests" is a request; the workspace is an enforcement point:

DEFAULT_PROTECTED = ("tests/**", "test_*.py", "*_test.py", "conftest.py",
                     ".git/**", "pyproject.toml", "setup.cfg")

An agent that can edit the thing that judges it has no failure mode left to report. The test suite includes the control — the same cheat with the guard removed, which succeeds — so the guard test is not passing for some other reason.

Refused edits are still recorded. "It never tried" and "it tried and was stopped" are very different reports, and an agent that discards its rejected attempts cannot be audited.

2. It spins

Propose an edit, tests fail, propose the same edit, tests fail, forever. Nothing stops it except the budget, and a budget is not a stopping condition — it is a timeout.

The fix needs a failure signature: an identity for this failure that survives noise.

def signature(self) -> str:
    if self.status == "pass":
        return "green"
    return "|".join(sorted(self.failing))   # test names, sorted

Not the raw output — that carries timings, temp paths and ordering, so two identical failures look different and the detector never fires. Two consecutive attempts with the same signature is a loop, and the run ends with no_progress.

report = run_agent(ws, runner, repeating_proposer(useless), max_attempts=50, patience=2)

assert report.outcome == "no_progress"
assert report.test_runs <= 4        # not 51

3. It declares victory it did not earn

A suite that could not run — collection error, syntax error, timeout — is not a suite with zero passes. They are different facts and they lead to different moves:

@property
def usable(self) -> bool:
    """False when the run tells us nothing."""
    return self.status in ("pass", "fail")

An agent that conflates them starts "fixing" tests that never executed. If the baseline run is unusable, this one stops immediately with unrunnable and touches nothing — there is no signal to work against, so any edit is a guess.

---

Four reasons to stop, and it always says which

outcome	meaning
fixed	the suite is green, and it got there by editing source
no_progress	the same failure twice running, or repeated attempts at protected files
budget	attempts exhausted while still making progress — the last resort
gave_up	the proposer had nothing left to try
unrunnable	the suite does not run, so nothing can be evaluated

budget should be rare. A loop that only ever ends on the budget has no stopping condition, it has a timer.

---

What it does when an edit makes things worse

Two checks, both automatic:

if not result.usable:                      # the edit broke the suite
    ws.revert_last()

regressed = result.passed < report.final.passed
if regressed and revert_on_regression:     # fewer tests pass than before
    ws.revert_last()

Rolling back beats reasoning about wreckage. The alternative — leaving a broken edit in place and asking the model to fix its own damage — is how a single bad step turns into a session that ends with the repo worse than it started.

---

The workspace is the security boundary

def resolve(self, rel: str) -> Path:
    target = (self.root / rel).resolve()
    if target != self.root and self.root not in target.parents:
        raise OutsideWorkspace(...)
    return target

The check is made after resolve(). Checking the string first is the version that looks correct and is not: ../, an absolute path and a symlink pointing outside are three different attacks and one line catches all of them.

And by default the agent works on a copy:

ws = Workspace.clone(source, scratch)      # the original is never touched

--in-place exists. It is not the default.

---

Swapping in a real model

The proposer is one callable:

def model_proposer(ws: Workspace, result: TestResult, attempt: int) -> Proposal | None:
    prompt = build_prompt(ws.files(), result.output)
    reply = call_your_model(prompt)          # NVIDIA NIM, Ollama, anything
    return parse_edit(reply)                 # or None to give up

run_agent(ws, PytestRunner(timeout=120), model_proposer, max_attempts=10, patience=2)

The guards, the rollback, the signature comparison and the four outcomes never learn what a model is. That is deliberate: the part that keeps the agent safe should not depend on the part that makes it clever.

The shipped default proposer cannot fix anything, on purpose. It makes python -m autocoder run end to end with no key, and it makes the default experience an honest one — the loop reports no_progress after two attempts instead of pretending.

---

Testing an agent without a model

Every test but one uses ScriptedRunner, a deterministic stand-in that reads the actual files:

def verdict_from_source(files: dict[str, str]) -> TestResult:
    src = files.get("target.py", "")
    if "a + b" in src:
        return TestResult(status="pass", passed=1)
    return TestResult(status="fail", passed=0, failed=1, failing=("tests/test_target.py::test_target",))

Reading the files is what keeps the fake honest — an agent cannot satisfy it by doing nothing. And one test shells out to real pytest end to end, so the stand-in is a convenience rather than the only thing that has ever been exercised.

---

Layout

src/autocoder/
  workspace.py   sandboxed edits, protected paths, rollback  - the security boundary
  runner.py      pytest subprocess + parsing; "could not run" is its own status
  agent.py       the loop, and the four reasons it stops
  __main__.py    python -m autocoder
tests/           24 tests, including the control for every guard

Licence

MIT