test(e2e): add coexistence suite, --repo mirroring, and fix two suite defects

[David Koleczek (DavidKoleczek)]

What this does

Adds end-to-end coverage for a guarantee that previously had none, and fixes two pre-existing defects in the e2e suite found while running it.

Test infrastructure only. No engine, CLI, wrapper, or protocol changes.

Why

0.15.0 established that amplifier-agent operates from its own ~/.amplifier-agent and leaves amplifier-app-cli's ~/.amplifier strictly alone. Nothing in tests/e2e/ asserted that. Every existing suite runs in a container where app-cli was never installed, so ~/.amplifier is nearly empty and the interesting case is never exercised. On a real machine that tree holds app-cli's live module clones.

The coexistence suite

Installs amplifier-app-cli inside the DTU, primes its bundle cache so the tree holds 28 real clone directories, records the state of ~/.amplifier, exercises amplifier-agent hard (run, doctor, config show, update --check, skills list, modes list, cache clear, then a second real turn), records again, and asserts nothing changed.

Five tests:

test_agent_does_not_touch_app_cli_tree       before/after comparison, 3789 files
test_app_cli_still_works_after_agent_runs    app-cli runs again afterwards
test_remote_skill_clones_stay_out_of_...     agent skill clones land in its own tree
test_isolation_guard_actually_fires          doctor's guard is not vacuous
test_clone_dirs_are_independent_not_aliased  colliding names are separate storage

app-cli is installed from the suite's own conftest.py rather than the DTU profile's setup_cmds, so only this suite pays the download.

Notes on two assertions that are less obvious than they look:

• The comparison uses path, size and mtime rather than content hashes. app-cli's cache is large, and a content change with identical size and mtime is not a realistic failure mode for git clones. The comparison was checked against out-of-band mutations to confirm it detects a new file, an mtime-only change, and an empty directory rather than always returning empty.
• The isolation guard is exercised via AMPLIFIER_AGENT_FOUNDATION_HOME, not by unsetting AMPLIFIER_HOME. amplifier_agent_lib/__init__.py rebinds that variable at package import, so the unset branch cannot be reached from a subprocess and a test written that way would pass vacuously.

--repo for the harness

The harness could only mirror amplifier-agent, plus amplifier-core and amplifier-foundation when dirty. A dependency whose changes needed exercising in the DTU had no way in, and a clean checkout on a branch was skipped because the criterion was working-tree dirtiness rather than "differs from upstream".

--repo <name>[@<ref>] on up/run/refresh mirrors any additional repo and injects its url_rewrites rule. Rules go into the staged temp copy of the profile, so the checked-in profile is never mutated and a run without the flag copies it byte-identically. Source repos are never mutated: resolve and fetch run inside a throwaway clone. Pushes only ever target the local Gitea container.

Two pre-existing defects fixed

Port collision. raw_capture and shadowing both hardcoded 9098. raw_capture's fixture is session-scoped, so it holds the port for the entire run and its teardown does not fire until the end; collection order puts it first, so shadowing could never bind. Deterministic rather than flaky: each suite passed alone, failed together, reproduced identically three times. Ports now live in tests/e2e/framework/ports.py naming each port and its owner, and shadowing moves to 9097. raw_capture's teardown also used a pkill pattern that could match its own argv and kill itself, orphaning the server; the bracketed-digit idiom shadowing already used is promoted into that module and shared.

Streaming timing assertion. It measured total elapsed between first and last content frame against a 0.5s floor. That scales with how much the model says, which the test does not control, so a genuinely streamed short response failed: six frames spanning 0.428s, an 86ms mean gap, reported as buffered.

It now measures the largest gap between consecutive frames. A streamed response has at least one real pause; a buffered dump has every gap at the measurement floor. That does not depend on response length. Threshold 50ms, about 6x below the slowest healthy observation (299ms) and 38x above the 1.3ms floor, chosen from repeated measurement. Max rather than mean because a leading burst of frames is normal and drags the mean down while the max stays healthy.

One thing to look at

streaming-mode now asserts SSE structure only, with no timing assertion.

It intermittently emits all content frames within a few milliseconds at the end of an 8-10 second turn, roughly 3 times in 12 observations. The connection opens promptly in those runs (first byte ~57ms), so the transport is not stalling, and streaming-plain over the same model and provider was clean in 9 of 9.

The cause is not established. The plain path being stable against the same API suggests a difference between the two paths rather than the API itself, but that has not been traced, so the case comment records the measurements and asserts no cause.

What this gives up is stated in the code: nothing now asserts streaming behavior on the active-mode path, and the suite docstring was corrected because it previously claimed coverage that no longer holds. Swapping the case back to "stream" restores it once the burst is understood.

Verification

make check                     ruff clean, 154 files formatted, pyright 0 errors
full cold DTU run              65 passed, 4 skipped in 334.81s, exit 0

The 4 skips are the vllm suite self-skipping with no server configured. Before this branch the same run gave 59 passed, 4 skipped, 1 error.

Confirmed against merged main, including the two follow-up fixes that landed with #142, so the before/after comparison covers the ChatGPT OAuth token copy-forward and the AMPLIFIER_AGENT_HOME injection.