test(e2e): add coexistence suite, --repo mirroring, and fix two suite defects

.amplifier/digital-twin-universe/profiles/e2e.yaml

@@ -47,8 +47,10 @@ url_rewrites:
   default_match_mode: boundary
   rules:
     - { match: github.com/microsoft/amplifier-agent, target: "${GITEA_URL}/admin/amplifier-agent" }
-    # Add amplifier-core / amplifier-foundation rules here to redirect them too when
-    # their working trees are being tested.
+    # This is the only STATIC rule. Additional rules are injected at launch time, one per
+    # `--repo <name>[@<ref>]` passed to the harness, into a temp copy of this file -- this
+    # checked-in profile is never modified at runtime. See
+    # tests/e2e/framework/dtu_manager.py::_profile_with_extra_rules.
 
 provision:
   files:

docs/E2E_TESTING.md

@@ -139,6 +139,37 @@ unknown feature name fails loud with the valid list.
 update for CLI iteration, but it leaves `serve` broken (provider-module note above), so HTTP tests
 need a full `run` / `up`.
 
+### Testing against extra repos
+
+By default only `amplifier-agent` (plus a dirty `amplifier-core` / `amplifier-foundation`) reaches
+the DTU. Anything else amplifier-agent depends on is fetched from real GitHub. `--repo` extends
+that set: it mirrors an additional repo into Gitea *and* injects the matching `url_rewrites` rule,
+so the DTU actually installs your version instead of the GitHub one. It is available on `up`, `run`
+and `refresh`, and is repeatable.
+
+```bash
+# your local amplifier-bundle-skills checkout, working tree and all
+uv run python tests/e2e/framework/cli.py run --repo amplifier-bundle-skills
+
+# a specific branch, ignoring whatever is in the working tree
+uv run python tests/e2e/framework/cli.py run --repo amplifier-bundle-skills@my-branch skills
+
+# a repo you have no local checkout of, cloned from GitHub at a ref (default main)
+uv run python tests/e2e/framework/cli.py up --repo amplifier-bundle-modes@v2
+
+# non-microsoft owner, and more than one repo at once
+uv run python tests/e2e/framework/cli.py run --repo someorg/their-bundle --repo amplifier-bundle-skills
+```
+
+The value is `[owner/]name[@ref]`. A bare name implies owner `microsoft`. The split is on the last
+`@`, so a ref containing `/` works. `--repo` is consumed by the harness and is never forwarded to
+pytest, so it can sit anywhere in the command line.
+
+The set of repos is recorded in the warm-DTU state file, because rewrite rules are baked into the
+container at launch. `run --skip-setup` with a different `--repo` set warns that the running DTU
+does not match what you asked for and that a full `run` / `up` is needed. `refresh` with no `--repo`
+re-mirrors exactly what the DTU was provisioned with.
+
 A normal `uv run pytest` (without the harness) still stays green, but that is now a weaker
 statement than it sounds: `tests/` contains only `tests/e2e/`, so a plain `pytest` run
 self-skips every collected test when `amplifier-digital-twin` is absent or no warm DTU
@@ -154,8 +185,33 @@ or for actually running this harness.
    redirect `github.com/microsoft/amplifier-agent` to that Gitea mirror, so
    `uv tool install --from git+...amplifier-agent` inside the DTU pulls your local tree.
 3. Only `amplifier-agent` is mirrored by default. `amplifier-core` / `amplifier-foundation` are
-   additionally snapshotted when their working trees are dirty (add matching `url_rewrites` rules
-   to extend redirection to them).
+   additionally snapshotted when their working trees are dirty. Mirroring alone changes nothing
+   inside the DTU, so those two still resolve from GitHub until a rewrite rule exists for them; the
+   harness prints a warning naming any repo in that state.
+4. `--repo <name>[@<ref>]` adds a repo to both halves at once: it is mirrored *and* gets a rewrite
+   rule injected into a temp copy of the profile at launch. The checked-in profile is never
+   modified at runtime.
+
+Where the content of an extra repo comes from:
+
+```
+local checkout in the workspace, no @ref  -> working-tree snapshot (same as amplifier-agent)
+local checkout in the workspace, w/ @ref  -> that committed ref; the working tree is ignored
+no local checkout                         -> cloned from GitHub at @ref (default main)
+```
+
+Two properties of this worth knowing:
+
+- Pushing only ever targets the local Gitea container. Nothing is ever pushed to GitHub. GitHub is
+  touched read-only, and only in the third case above, to clone or fetch a repo you have no local
+  copy of.
+- Your source repo is never mutated, in any case. For `@ref` on a local checkout the harness clones
+  it into a temp dir first and resolves or fetches the ref inside that clone, so no git command
+  ever runs against your checkout.
+
+Whatever ref you pick lands in the mirror as `main`. That is deliberate: a bundle that references
+`...@main` resolves to the mirror's `main`, so pointing a `--repo` at a PR branch tests that branch
+without editing any `@main` reference.
 
 Everything about *how* amplifier-agent is installed lives in
 `framework/provisioning/install-amplifier-agent.sh` and `host-config.json`. Change the install
@@ -231,6 +287,39 @@ pushing from a suite-local `conftest.py` fixture that returns the in-DTU paths.
 case's `cwd` when the behavior under test keys off the launch directory. See `suites/skills/` for a
 worked example (seeds a skill into a launch-dir `.amplifier/skills/` and a configured location).
 
+### The `coexistence` suite
+
+Every other suite runs in a container where amplifier-app-cli was never installed, so
+`~/.amplifier` is nearly empty. That is the easy case. `coexistence` tests the case a real
+user is in: both applications installed side by side, with app-cli's live module clones
+sitting in `~/.amplifier`.
+
+What it proves is `docs/spec/foundation-cache-ownership.md`: amplifier-agent operates
+entirely from `~/.amplifier-agent` and leaves app-cli's tree strictly alone. It records
+`~/.amplifier` (path + size + mtime per file, plus the directory set), exercises
+amplifier-agent hard, records it again, and asserts the two are identical. It then runs
+app-cli again to confirm it still works, checks that `doctor`'s foundation-isolation guard
+actually fires when isolation is broken, and confirms the two cache roots are separate
+storage rather than two names for one directory.
+
+```bash
+uv run python tests/e2e/framework/cli.py run coexistence
+```
+
+It is slower than the other suites, and deliberately so. The suite installs
+amplifier-app-cli inside the DTU on demand from its own `conftest.py` rather than from the
+DTU profile's `setup_cmds`, so a normal `run` of anything else never pays for the download.
+The first run in a container also has to prime app-cli's bundle cache, which clones its
+whole module set. Progress is logged as it goes so a slow run is not mistaken for a hang.
+
+All five tests run by default. The one covering remote skill clones used to skip, because
+upstream `tool-skills` hardcoded `~/.amplifier/cache/skills` as its clone root regardless
+of `AMPLIFIER_HOME`; microsoft/amplifier-bundle-skills#61 fixed that and has merged, so a
+stock DTU now installs a `tool-skills` that honours `AMPLIFIER_HOME` and the test exercises
+the real thing. It still probes for the fix rather than assuming it, so a DTU provisioned
+from an older skills checkout skips with a clear reason instead of failing as if
+amplifier-agent had regressed.
+
 ### Tests for features that do not exist yet
 
 Mark them `@pytest.mark.xfail(reason="...", strict=True)`. The test still runs (it really hits the

tests/e2e/conftest.py

@@ -22,7 +22,7 @@
 # tests/e2e/, is their shared parent).
 sys.path.insert(0, str(Path(__file__).resolve().parent))
 
-from framework import dtu, state
+from framework import dtu, ports, state
 
 
 def pytest_configure(config: pytest.Config) -> None:
@@ -59,17 +59,19 @@ def dtu_id(e2e_state: dict[str, Any]) -> str:
 def server(dtu_id: str) -> Generator[dict[str, str], None, None]:
     """Start the amplifier-agent HTTP server INSIDE the DTU once for HTTP cases.
 
-    Launches ``serve chat-completions`` bound to 0.0.0.0:9099 (so curl-from-inside
-    on localhost works), then polls ``/v1/models`` until it answers 200 or ~60s pass.
-    Yields the base_url + bearer token. Best-effort pkill on teardown.
+    Launches ``serve chat-completions`` bound to 0.0.0.0 on
+    ``ports.SHARED_SERVER_PORT`` (so curl-from-inside on localhost works), then polls
+    ``/v1/models`` until it answers 200 or ~60s pass. Yields the base_url + bearer
+    token. Best-effort pkill on teardown.
     """
-    base_url = "http://localhost:9099"
+    port = ports.SHARED_SERVER_PORT
+    base_url = f"http://localhost:{port}"
     token = "local-dev-secret"
 
     start = (
         "mkdir -p /root/e2e && "
         "nohup amplifier-agent serve chat-completions "
-        "--bind 0.0.0.0 --port 9099 --api-key local-dev-secret "
+        f"--bind 0.0.0.0 --port {port} --api-key {token} "
         ">/root/e2e/serve.log 2>&1 &"
     )
     dtu.exec_json(dtu_id, ["bash", "-lc", start])

tests/e2e/framework/cli.py

@@ -7,6 +7,9 @@
              Optionally scope to one or more features: ``run skills``, ``run run modes``.
     down     Tear down the DTU instance (leaves Gitea running).
 
+``up``/``run``/``refresh`` accept a repeatable ``--repo NAME[@REF]`` that mirrors an
+additional repo into Gitea and redirects it inside the DTU. See docs/E2E_TESTING.md.
+
 Not installed as a console script; runs directly via uv run.
 """
 
@@ -87,26 +90,42 @@ def cli() -> None:
     """amplifier-agent e2e harness."""
 
 
+_REPO_OPTION = click.option(
+    "--repo",
+    "repos",
+    multiple=True,
+    metavar="NAME[@REF]",
+    help=(
+        "Mirror an ADDITIONAL repo into Gitea and redirect it inside the DTU. "
+        "NAME is a bare repo (owner defaults to microsoft) or owner/repo; @REF pins a "
+        "git ref. Repeatable, e.g. --repo amplifier-bundle-skills --repo foo@my-branch."
+    ),
+)
+
+
 @cli.command()
-def up() -> None:
+@_REPO_OPTION
+def up(repos: tuple[str, ...]) -> None:
     """Provision a fresh warm DTU (destroys any existing aa-e2e) and print the state JSON."""
     _preflight()
-    new_state = dtu_manager.provision()
+    new_state = dtu_manager.provision(repos)
     click.echo(json.dumps(new_state, indent=2))
 
 
 @cli.command()
-def refresh() -> None:
+@_REPO_OPTION
+def refresh(repos: tuple[str, ...]) -> None:
     """Re-push local repos and reinstall in place inside the warm DTU."""
-    dtu_manager.refresh()
+    dtu_manager.refresh(repos)
     click.echo("refreshed")
 
 
 @cli.command(context_settings={"ignore_unknown_options": True})
 @click.option("--skip-setup", is_flag=True, help="Skip the Gitea push + fresh rebuild; run against the DTU as-is.")
 @click.option("--ephemeral", is_flag=True, help="Tear down the DTU after the run.")
+@_REPO_OPTION
 @click.argument("args", nargs=-1, type=click.UNPROCESSED)
-def run(skip_setup: bool, ephemeral: bool, args: tuple[str, ...]) -> None:
+def run(skip_setup: bool, ephemeral: bool, repos: tuple[str, ...], args: tuple[str, ...]) -> None:
     """Push latest code, provision a fresh DTU with it, then run the e2e pytest suite.
 
     By default every run re-mirrors the working tree to Gitea and rebuilds the DTU clean
@@ -115,7 +134,8 @@ def run(skip_setup: bool, ephemeral: bool, args: tuple[str, ...]) -> None:
 
     Optionally scope the run to one or more features (directories under tests/e2e/suites/),
     e.g. ``cli.py run skills`` or ``cli.py run run modes``. Any remaining args (flags, `-k`
-    expressions, explicit node ids) pass straight through to pytest.
+    expressions, explicit node ids) pass straight through to pytest. ``--repo`` is consumed
+    here and never forwarded to pytest.
     """
     _preflight()
 
@@ -124,9 +144,10 @@ def run(skip_setup: bool, ephemeral: bool, args: tuple[str, ...]) -> None:
     if skip_setup:
         if not dtu_manager.is_warm():
             raise click.ClickException("no warm DTU and --skip-setup set; run `up` first")
+        dtu_manager.warn_repo_mismatch(repos)
         log("run: --skip-setup; using existing warm DTU as-is")
     else:
-        dtu_manager.provision()
+        dtu_manager.provision(repos)
 
     if features:
         targets = [f"tests/e2e/suites/{feature}" for feature in features]