End-to-end test of owloop go on xyz-data-website failed to complete the task

[caoergou]

Issue: End-to-end test of owloop go on xyz-data-website failed to complete the task

Type: Bug / reliability issue
Severity: High — blocks autonomous use of owloop go for real tasks
Discovered during: Manual end-to-end test of owloop on xyz-data-website, 2026-07-07/08
Owloop version under test: main around commit dcd69b6 plus local fixes on branch fix/ignore-owloop-paths-in-dirty-check

---

Summary

Running owloop go "<complex prompt>" against xyz-data-website did not result in a completed, verifiable implementation. Instead, the loop either stalled before doing meaningful work or produced an implementation that did not match the user's explicit requirements. The round-trip surfaced several independent but critical failures in owloop itself.

The user asked for:

1. An end-to-end owloop run on xyz-data-website.
2. Start with owloop go + prompt.
3. The task should be derived from the front-end and back-end code and result in a schema assigned to a back-end interface.
4. A timeout of 3600 s.

What actually happened:

• The generated spec/prompt was too generic and did not encode the concrete task (schema extraction from front-end/back-end code).
• The run used the default front-end timeout (300 s) instead of the requested 3600 s.
• The worktree was created but the loop appeared to stall at iterations: 0.
• During manual recovery, acceptance-criteria commands such as uv run pytest mutated uv.lock, which was listed in ## Exclusions, causing the verification gate to fail even though the mutation was a side effect of the test runner, not the agent.
• .owloop/ metadata was at risk of being committed because the directory was created lazily and the commit path used git add -A followed by git reset -- .owloop/.

---

Observed failures

1. User goal is lost after spec generation

File: src/owloop/commands/go.py:84 → src/owloop/commands/run.py → src/owloop/engine.py:_build_prompt_with_context

owloop go receives the user's natural-language goal, feeds it to SpecGenerator, then starts the engine with parameters derived from CLI options. The original goal is not forwarded to the build iterations. The agent only sees the generated spec and a hard-coded BUILD_PROMPT (src/owloop/engine.py:49).

Consequence: if the generated spec is vague or misses nuance from the original prompt, the agent has no way to recover that context. In this test, the prompt "according to front-end and back-end code, assign a schema to the back-end interface" was reduced to something far more generic, and the agent began doing unrelated refactoring instead of schema extraction.

2. Front-end timeout is hard-capped at 300 s

Files: src/owloop/adapters.py, CLI option parsing in src/owloop/cli_options.py

The user explicitly requested a 3600 s timeout. The CLI/options layer appears to cap or default to 300 s for foreground tool calls, and this cap is passed to the adapter. Long-running commands (e.g., uv sync, model downloads, full test suites) are aborted before completion, leaving the worktree in an inconsistent state.

3. Verification gate treats dependency-manager side effects as agent violations

Files: src/owloop/verification.py, src/owloop/spec_queue.py

A common acceptance criterion such as:

- [ ] `uv run pytest` → 28 passed

causes uv to rewrite uv.lock. If uv.lock is listed in ## Exclusions, the old verifier restored it silently (hiding the real test result) or, in the newer code, flagged the iteration as failed. There was no way to express "the command output should contain 28 passed" as a success condition; only exit-code success was supported.

4. Exclusion list parsing was too naive

File: src/owloop/spec_queue.py:get_spec_exclusions

Prose-style exclusions such as:

- Do NOT modify `backend/uv.lock` or `backend/pyproject.toml`

were not parsed for backtick-quoted paths. The function either treated the whole bullet as a single invalid glob or ignored it, so excluded files were not actually protected.

5. Exclusion enforcement only unstaged files, did not restore content

File: src/owloop/verification.py / src/owloop/engine.py:_commit_iteration

The old commit path did:

git add -A
git reset --quiet -- .owloop/

This prevented .owloop/ from entering the commit, but it did nothing for excluded project files that had been modified by the agent or by acceptance-criteria commands. Those modifications remained in the working tree and could affect the next iteration or be manually committed later.

6. .owloop/ directory could be created too late and tracked by git

File: src/owloop/engine.py

If the main repo did not already have a .owloop/ directory, the engine created it on demand. Depending on .gitignore state, git add -A could pick it up before the reset. The safer fix is to create .owloop/ up front and exclude it from the add entirely.

7. Spec linter did not reject exit-code-swallowing acceptance criteria

File: src/owloop/spec_linter.py

Acceptance criteria such as:

- [ ] `ruff check app.py | tail -1` → no errors
- [ ] `false || true` → ok

passed linting even though they hide the real exit code, making failures invisible to the verification gate.

8. The loop stalled at iteration 0

Log: .owloop/logs/session_latest.json

{
  "session_id": "0fb03f69",
  "branch": "owloop/20260707-0fb03f69",
  "path": "/home/xyz/EricCao/projects/owloop-owloop-wt/owloop-20260707-0fb03f69",
  "started_at": "2026-07-07T17:27:53.353416",
  "status": "running",
  "iterations": 0,
  "tokens_used": 0,
  "elapsed_seconds": 0.0,
  "current_spec": null
}

The worktree was created but the engine never progressed to the first iteration. The events.jsonl only records dirty_workspace_warning, worktree_prompt, and worktree_creating. Root cause was not fully diagnosed, but it correlates with the dirty-workspace check incorrectly flagging .owloop/ metadata as uncommitted changes.

---

Why owloop could not complete the task

The failures compound:

1. Wrong problem definition: the agent never received the user's real intent, so it could not produce the right implementation.
2. Wrong time budget: long-running verification was killed after 300 s, so even a correct agent could not finish.
3. False verification failures: legitimate acceptance-criteria side effects were treated as agent errors, causing rollbacks.
4. Unprotected exclusions: files the user explicitly wanted untouched were modified and not restored.
5. Metadata pollution: the loop's own bookkeeping files risked being committed alongside real changes.
6. Stall on start: in at least one attempt, the engine never began iterating.

Because of these issues, the loop could not reach a verified <promise>DONE</promise> state for the requested task.

---

Local fixes already drafted

The following changes are present in the working tree on fix/ignore-owloop-paths-in-dirty-check and their tests pass (107 passed in 9.05s). They address most of the items above but have not been reviewed or merged:

• src/owloop/engine.py: create .owloop/ early; exclude it from git add with :.owloop/ instead of add-then-reset.
• src/owloop/spec_queue.py: parse backtick-quoted paths in ## Exclusions; add expect_contains to AcceptanceCriterion.
• src/owloop/verification.py: detect excluded files modified by agent or AC commands, restore them, and fail the gate.
• src/owloop/spec_linter.py: reject acceptance criteria that swallow exit codes.
• templates/spec-template.md: updated guidance.

Still not addressed by the local fixes:

• Forwarding the original goal into iteration prompts so the agent retains the full user intent.
• Removing or raising the 300 s foreground timeout cap.
• Diagnosing and fixing the iteration-0 stall.

---

Suggested acceptance criteria for closing this issue

A verified fix should satisfy:

• owloop go "derive a JSON schema for the back-end API from the front-end forms in xyz-data-website" --timeout 3600 generates a spec whose requirements explicitly mention front-end/back-end code analysis and schema assignment.
• The build agent's prompt includes the original user goal in addition to the generated spec.
• A 3600 s timeout can be passed through the CLI and is honored by the adapter.
• Acceptance criteria can express substring expectations (e.g., → contains 28 passed).
• Acceptance-criteria commands that legitimately rewrite excluded lockfiles do not fail the gate.
• Files listed in ## Exclusions are restored to HEAD when modified by the agent.
• .owloop/ metadata never appears in commits.
• The engine does not stall at iteration 0 on a clean or dirty workspace.

---

Attachments

• Session log: .owloop/logs/session_latest.json
• Event log: .owloop/logs/events.jsonl
• Local report: .owloop/issues/owloop-e2e-test-failures-20260708.md