fix(agents): idempotency discipline for git-ops (context-truncation thrash) - #317
Merged
Merged
Conversation
…n thrash) Production eval evidence (5 runs, fast-tier model, enable_long_context=false) showed git-ops re-running `git clone --branch main --single-branch <same URL>` up to 8x and `git ls-remote --symref <same URL>` up to 7x within a single session (one run: 30 bash calls, only 14 unique). This is the signature of context truncation losing the memory that discovery/setup work already completed, inflating 'fast' sessions to 809-1168s. Adds a compact 'Idempotency Discipline' checklist positioned right after the existing Git Safety Protocol section (near the top of operational guidance, before the command references) so it survives truncation: check disk/remote state before cloning, run remote-discovery commands at most once per remote per session, and treat an about-to-repeat command as a signal to check repo state instead of re-running it. Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Collaborator
Author
|
Maintainer admin-merge basis (documented per team policy):
Squash-merging now. |
This was referenced Aug 27, 2026
Brian Krabach (bkrabach)
added a commit
that referenced
this pull request
Aug 28, 2026
…idempotency section (evidence refuted) (#328) A payload-level investigation found that three recent changes were built on a measurement artifact: analyses counted the same events 2-4x across duplicate snapshot copies of session files (proven by identical tool_call_id + nanosecond timestamps across "duplicates"). Deduped, the phenomena each change was built to address vanish. This reverts all three on that evidence. 1. Remove hooks-dedupe module entirely (PR #323 as hooks-tool-dedupe, renamed in PR #326). Deleted modules/hooks-dedupe/ and its `hooks:` entry in behaviors/agents.yaml. Deduped, the "same-batch duplicate reads" it coalesces do not exist: 0 across 314 sessions / 10,320 reads. Measured real value was ~19k tokens (~$0.003) across 5 runs against a claimed 16.4% savings -- a ~4,800x overstatement. Kept: tests/test_hook_module_classification.py (added by #326). It guards a real amplifier-core loader fragility (name-based module-type guessing misclassifying a `hooks-*` module as `tool`) that is unrelated to whether hooks-dedupe itself exists. Removed only the hooks-dedupe specific references: the hardcoded `assert "hooks-dedupe" in HOOK_MODULE_IDS` and the `test_hooks_dedupe_passes_real_validation` end-to-end test (and its now-unused `ModuleValidationError` import). The general, dynamically parametrized `test_hook_module_classifies_as_hook` test is untouched and still covers every remaining `hooks-*` module. 2. Remove PR #320's "BATCH YOUR DELEGATIONS" guidance block from the delegate tool's description in modules/tool-delegate/amplifier_module_tool_delegate/__init__.py, restoring the original "- Launch multiple agents concurrently when tasks are independent" line it replaced. A clean same-commit 5v5 A/B measured the guidance as inert (treatment waves median 4 vs control median 3) -- the earlier apparent win was a version confound -- while it cost ~175 tokens on every single request. PR #320's two context-file edits (context/agents/multi-agent-patterns.md, context/agents/delegation-instructions.md) are untouched; only the scope named above is in this revert. PR #327's own deletions in this same string (the CRITICAL/ALWAYS/NEVER preamble and the "DEFAULT TO DELEGATION" line) are also untouched -- they stay removed. 3. Remove PR #317's "## Idempotency Discipline" section from agents/git-ops.md. Its evidence -- "git-ops re-ran identical clone x8 / ls-remote x7 under context truncation" -- is refuted: deduped session data shows 8 clones of 8 *different* repos, each cloned once, with per-repo ls-remote calls. The thrash it was written to prevent never happened. Verified: full suite green (uv run pytest tests/ -q: 1640 passed, 1 skipped -- down from 1642 by exactly the two hooks-dedupe-specific test instances removed in (1), both accounted for). python_check clean on all touched files (only pre-existing, unrelated ruff warnings remain in tool-delegate's __init__.py, none introduced by this change). Repo-wide grep confirms modules/hooks-dedupe/ is gone with no dangling functional references (three remaining mentions of hooks-tool-dedupe/hooks-dedupe are historical narrative inside the kept regression test's docstring/assert message, explaining why that general test exists). 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-authored-by: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Collaborator
Author
|
CORRECTION: this PR's evidentiary basis has been refuted. Deduped payload-level re-analysis shows the "git clone x8 / ls-remote x7 thrash" was 8 DIFFERENT provider repos cloned once each plus per-repo remote checks — legitimate work, not context-truncation thrash. The original count came from duplicate snapshot copies of session event files being counted 2-4x (verified via identical tool_call_id + timestamps). The Idempotency Discipline section is being removed in #328. The mid-PR observation that enable_long_context config is provider-level (not routing-matrix) stands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agents/git-ops.md, positioned right after the existing "Git Safety Protocol" section (near the top of operational guidance, before the command references) so it survives context truncation.openai_improvement-bq0: 5 production eval runs (fast-tier model,enable_long_context=false) showed git-ops re-runninggit clone --branch main --single-branch <same URL>up to 8x andgit ls-remote --symref <same URL>up to 7x within a single session (one run: 30 bash calls, only 14 unique) — the signature of context truncation losing the memory that setup work already completed. Sessions ran 809–1168s despite the "fast" label.Added text (exact)
Inserted between the existing "Git Safety Protocol" and "Common Git Commands" sections (
agents/git-ops.md:124-133on this branch).Test plan
enable_long_context=falsematrix entry that produced the original evidence, to confirm the repeat-clone/repeat-ls-remote pattern no longer recurs.Recommendation for routing-matrix maintainers (informational — no code change here)
I looked in
microsoft/amplifier-bundle-routing-matrix(shallow clone ofmain) for where thefastrole setsenable_long_context=false, to cite it here. Honest finding: the literal keyenable_long_contextdoes not appear anywhere in that repository — not in any of the eight shipped matrices (routing/anthropic.yaml,balanced.yaml,copilot.yaml,economy.yaml,gemini.yaml,ollama.yaml,openai.yaml,quality.yaml), not inbehaviors/routing.yamlorcontext/routing-instructions.md, and not in thehooks-routingmodule source (modules/hooks-routing/amplifier_module_hooks_routing/{__init__,matrix_loader,resolver,resolver_class}.py). Everyfast:role block in every shipped matrix is configless (candidates only, noconfig:block at all); the only per-candidateconfig:keys used anywhere in the repo arereasoning_effort(andpriority, internally).matrix_loader.validate_matrix_configis explicitly "closed on values, OPEN on keys" — it validates candidateconfig:keys against whateverconfig_fieldsthe installed provider module reports, meaningenable_long_contextis a provider-level config field, not something the routing-matrix repo itself declares or sets.The
enable_long_context=falsevalue cited in the eval evidence traces to this CI harness's own run summary (_ci_eval_summary.jsoninopenai-evals-team-ci), where thefast-role spawn foranchors-amp-dev:git-opsresolves toprovider: luna, model: gpt-5.6-lunawithconfig: {enable_long_context: "false", ...}— i.e. it's set by the eval harness's per-role provider config (or the underlying provider module it resolves to), not by a literal setting insideamplifier-bundle-routing-matrix.Recommendation: whoever owns that per-role provider config (eval harness config, or the
luna/fast-tier provider module ifenable_long_contextis a first-class field there) should re-evaluate defaultingenable_long_context=falsefor agents likegit-opsthat accumulate long, repetitive bash command histories across a session — this idempotency fix reduces the symptom (repeated commands), but the truncation-thrash evidence above suggests the root cause (losing session memory mid-run) would also benefit from being addressed at the context-window-management layer for operational agents.Generated with Amplifier