Skip to content

fix(bundle/cache): re-stamp agent source_path on warm-path cache hit (cross-installation FileNotFoundError) - #160

Open
Brian Krabach (bkrabach) wants to merge 1 commit into
microsoft:mainfrom
bkrabach:fix/agent-source-path-cross-install-cache
Open

fix(bundle/cache): re-stamp agent source_path on warm-path cache hit (cross-installation FileNotFoundError)#160
Brian Krabach (bkrabach) wants to merge 1 commit into
microsoft:mainfrom
bkrabach:fix/agent-source-path-cross-install-cache

Conversation

@bkrabach

Copy link
Copy Markdown
Contributor

Problem

Embedding amplifier-agent per docs/INTEGRATION.md's documented recipe
(load_and_prepare_cached -> inject_provider/inject_routing_matrix ->
Engine(turn_handler) -> boot -> submit_turn) from a fresh
installation
can raise FileNotFoundError on the very first turn, with no
code path other than the vendored library involved. Reported by a downstream
embedder integrating via docs/INTEGRATION.md (a .dot-pipeline adapter
module that hosts Engine as a node worker), traced back to this library
with a minimal, dot-runner-free reproduction below.

Traceback (minimal repro, docs/INTEGRATION.md's own embedding recipe, real API key)

Traceback (most recent call last):
  File "repro.py", line 63, in <module>
    asyncio.run(main())
  ...
  File "repro.py", line 25, in main
    handler = make_turn_handler(
              ^^^^^^^^^^^^^^^^^^
  File ".../site-packages/amplifier_agent_lib/_runtime.py", line 446, in make_turn_handler
    name: hydrate_agent_overlay(Path(entry["source_path"]))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/amplifier_agent_lib/spawn.py", line 110, in hydrate_agent_overlay
    text = agent_md_path.read_text(encoding="utf-8-sig")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '<other-install>/lib/python3.12/site-packages/amplifier_agent_lib/bundle/agents/explorer.md'

The path in the error belongs to a different, no-longer-current
installation of this same package/version -- not the one that raised.

Root cause

load_and_prepare_cached (src/amplifier_agent_lib/bundle/cache.py) keys its
on-disk warm-path cache
($AMPLIFIER_AGENT_HOME/cache/prepared/<aaa_version>/<sha256(bundle.md)>/prepared.pickle)
by (aaa_version, sha256(bundle.md content)) only. That key is scoped to
$AMPLIFIER_AGENT_HOME, which defaults to ~/.amplifier-agent -- per
user, not per installation.

bundle/loader.py's cold path enriches bundle.agents[name]["source_path"]
with an absolute path resolved against whichever installation happened to
run the cold prepare (ultimately derived from BUNDLE_DIR = Path(__file__).parent
in bundle/__init__.py). That absolute path is part of what gets pickled into
the cache artifact.

Any second installation of the same version with an unmodified vendored
bundle.md (a fresh uv tool install after an earlier one was removed and
reinstalled elsewhere, a dev checkout run alongside a packaged install, two
side-by-side venvs, a CI runner that keeps ~/.amplifier-agent across
ephemeral installs) warm-hits the same cache entry and gets back agent
source_path values pointing at the first installation's site-packages
tree. make_turn_handler reads that path directly
(hydrate_agent_overlay(Path(entry["source_path"]))), so the very first turn
on the second installation raises FileNotFoundError -- even though that
second installation ships the exact same vendored agents/*.md file, at its
own, different, path. resources.BUNDLE_DIR already documents "resolves
correctly whether running from source or installed in a wheel because it is
derived from __file__
" -- the pickled source_path is exactly the case
that guarantee doesn't reach, because it was resolved once, by a different
process, and then frozen into a cache keyed on something that doesn't capture
"which installation."

I've verified this concretely: build the cache from installation A, delete
A, install a fresh copy at a different path (B, same version, same
bundle.md), reuse the same $AMPLIFIER_AGENT_HOME -- B's first turn raises
exactly the FileNotFoundError above, pointing at A's now-deleted path.

This is not a packaging gap (every individual installation ships
agents/*.md correctly -- scripts/verify-wheel.py already guards that),
and it reproduces with zero application code beyond this library's own
documented embedding recipe, so it isn't a caller-side bug either.

Fix

bundle/cache.py: after every warm-path deserialize, re-stamp each agent's
source_path against this process's own AGENTS_DIR (also
__file__-derived, so always correct for whichever installation is actually
running) before returning the PreparedBundle -- mirroring
bundle/loader.py's own cold-path rule of only stamping a path it has
verified exists. No cache-key change, no invalidation, no re-clone/re-install
cost; only the one baked-in-per-process field is refreshed.

Added scripts/verify-cache-restamp.py (this repo's standalone-script
convention -- see scripts/verify-wheel.py's own rationale for why this
isn't a pytest test) covering: the restamp helper directly (stale path fixed,
unresolvable name left alone, no-op without an agents section), and an
end-to-end run of load_and_prepare_cached against a pickle built for a
since-deleted installation path, confirming the warm path now returns
readable paths instead of raising.

Note for maintainers (related, out of scope for this PR)

The same root gap (a per-user cache reused across installations that may not
share Python dependency state) can plausibly also produce
ModuleNotFoundError for a bundle-declared module's own pip dependency
(e.g. a provider module's SDK package) if the cold prepare that built the
cache ran in a different interpreter than the one warm-hitting it later --
install_deps only runs on the cold path. This PR fixes the concrete,
reproducible FileNotFoundError on vendored resources; the broader
dependency-install-vs-cache-sharing question may be worth its own design
discussion.

Testing

uv run scripts/verify-cache-restamp.py

passes all 4 checks against this branch.

The prepared-bundle cache (~/.amplifier-agent/cache/prepared/<aaa_version>/
<bundle_sha256>/prepared.pickle) is keyed only by (aaa_version,
sha256(bundle.md content)) -- a per-user key, not a per-installation one.
PreparedBundle.mount_plan["agents"][name]["source_path"] bakes in an
ABSOLUTE path resolved against whichever installation ran the cold prepare.

A second installation of the same version (e.g. a fresh 'uv tool install'
after an earlier one was removed, or a dev checkout alongside a packaged
install) warm-hits the same cache entry and gets back agent source_path
values pointing at the FIRST installation's site-packages tree. When that
location no longer exists, make_turn_handler's cold-path agent-overlay
hydration (hydrate_agent_overlay(Path(entry["source_path"]))) raises
FileNotFoundError on the very first turn -- even though the CURRENT
installation ships that exact vendored agents/*.md file at its own,
different, path.

Fix: after every warm-path deserialize, re-stamp each agent's source_path
against *this* process's own AGENTS_DIR (Path(__file__)-derived, so it is
always correct for whichever installation is actually running), mirroring
bundle/loader.py's own cold-path rule of only stamping a path verified to
exist.

Adds scripts/verify-cache-restamp.py (this repo's standalone-script
convention, not a pytest test -- see verify-wheel.py) covering the restamp
helper directly and an end-to-end warm-path run against a pickle built for
a since-deleted installation path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant