Skip to content

fix(agents): idempotency discipline for git-ops (context-truncation thrash) - #317

Merged
Brian Krabach (bkrabach) merged 1 commit into
mainfrom
fix/git-ops-idempotency
Aug 27, 2026
Merged

fix(agents): idempotency discipline for git-ops (context-truncation thrash)#317
Brian Krabach (bkrabach) merged 1 commit into
mainfrom
fix/git-ops-idempotency

Conversation

@bkrabach

Copy link
Copy Markdown
Collaborator

Summary

  • Adds a compact Idempotency Discipline checklist to agents/git-ops.md, positioned right after the existing "Git Safety Protocol" section (near the top of operational guidance, before the command references) so it survives context truncation.
  • Addresses work item openai_improvement-bq0: 5 production eval runs (fast-tier model, enable_long_context=false) showed git-ops re-running git clone --branch main --single-branch <same URL> up to 8x and git ls-remote --symref <same URL> up to 7x within a single session (one run: 30 bash calls, only 14 unique) — the signature of context truncation losing the memory that setup work already completed. Sessions ran 809–1168s despite the "fast" label.
  • Prompt/agent-definition change only — no Python touched.

Added text (exact)

## Idempotency Discipline

Disk and repo state are the source of truth — not your memory of what you already ran.

**Before ANY clone:** check whether the target directory already exists with the right remote (`git -C <dir> remote get-url origin`). If it matches, `git -C <dir> fetch` instead of re-cloning. Never clone the same URL twice in one session.

**Remote-discovery commands run at most once per remote per session.** `git ls-remote`, default-branch lookups, and similar discovery calls are one-shot — write the result into your working notes immediately and reuse it. Don't re-derive it.

**About to repeat a command you believe you already ran? Stop.** Check the filesystem/repo state first. Repeated identical commands are the signature of lost context, not a reason to run them again.

Inserted between the existing "Git Safety Protocol" and "Common Git Commands" sections (agents/git-ops.md:124-133 on this branch).

Test plan

  • N/A — prompt/agent-definition-only change (Markdown frontmatter + prose), no code paths to unit test.
  • Diff reviewed for minimality: 10 lines added, 0 removed, no restructuring of surrounding sections.
  • Recommend validating with a DTU/eval re-run against the same fast-tier/enable_long_context=false matrix entry that produced the original evidence, to confirm the repeat-clone/repeat-ls-remote pattern no longer recurs.

Recommendation for routing-matrix maintainers (informational — no code change here)

I looked in microsoft/amplifier-bundle-routing-matrix (shallow clone of main) for where the fast role sets enable_long_context=false, to cite it here. Honest finding: the literal key enable_long_context does not appear anywhere in that repository — not in any of the eight shipped matrices (routing/anthropic.yaml, balanced.yaml, copilot.yaml, economy.yaml, gemini.yaml, ollama.yaml, openai.yaml, quality.yaml), not in behaviors/routing.yaml or context/routing-instructions.md, and not in the hooks-routing module source (modules/hooks-routing/amplifier_module_hooks_routing/{__init__,matrix_loader,resolver,resolver_class}.py). Every fast: role block in every shipped matrix is configless (candidates only, no config: block at all); the only per-candidate config: keys used anywhere in the repo are reasoning_effort (and priority, internally). matrix_loader.validate_matrix_config is explicitly "closed on values, OPEN on keys" — it validates candidate config: keys against whatever config_fields the installed provider module reports, meaning enable_long_context is a provider-level config field, not something the routing-matrix repo itself declares or sets.

The enable_long_context=false value cited in the eval evidence traces to this CI harness's own run summary (_ci_eval_summary.json in openai-evals-team-ci), where the fast-role spawn for anchors-amp-dev:git-ops resolves to provider: luna, model: gpt-5.6-luna with config: {enable_long_context: "false", ...} — i.e. it's set by the eval harness's per-role provider config (or the underlying provider module it resolves to), not by a literal setting inside amplifier-bundle-routing-matrix.

Recommendation: whoever owns that per-role provider config (eval harness config, or the luna/fast-tier provider module if enable_long_context is a first-class field there) should re-evaluate defaulting enable_long_context=false for agents like git-ops that accumulate long, repetitive bash command histories across a session — this idempotency fix reduces the symptom (repeated commands), but the truncation-thrash evidence above suggests the root cause (losing session memory mid-run) would also benefit from being addressed at the context-window-management layer for operational agents.

Generated with Amplifier

…n thrash)

Production eval evidence (5 runs, fast-tier model, enable_long_context=false)
showed git-ops re-running `git clone --branch main --single-branch <same URL>`
up to 8x and `git ls-remote --symref <same URL>` up to 7x within a single
session (one run: 30 bash calls, only 14 unique). This is the signature of
context truncation losing the memory that discovery/setup work already
completed, inflating 'fast' sessions to 809-1168s.

Adds a compact 'Idempotency Discipline' checklist positioned right after the
existing Git Safety Protocol section (near the top of operational guidance,
before the command references) so it survives truncation: check disk/remote
state before cloning, run remote-discovery commands at most once per remote
per session, and treat an about-to-repeat command as a signal to check repo
state instead of re-running it.

Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@bkrabach

Copy link
Copy Markdown
Collaborator Author

Maintainer admin-merge basis (documented per team policy):

  • Author: this PR was authored by our own automated account. GitHub structurally forbids self-approval, so no self-review can satisfy branch protection here — an external human reviewer is not in the loop for this change.
  • Test evidence: 10-line agent-definition addition (no code). Root cause verified via fast-tier git-ops re-running identical clone (x8) and ls-remote (x7) under context truncation — idempotency discipline added to prevent this thrash. No new test suite changes required; behavior is instruction-level.
  • CI status at merge time: all 6 matrix jobs (ubuntu/windows × Python 3.11/3.12/3.13) + license/cla check green. See status checks on this PR.
  • Action: merging via maintainer admin override (gh pr merge --admin) at the explicit direction of the repository owner, following the same sanctioned pattern previously used for gemini#39 (self-authored, self-approval-impossible, explicit maintainer direction).

Squash-merging now.

@bkrabach
Brian Krabach (bkrabach) merged commit 835c558 into main Aug 27, 2026
7 checks passed
Brian Krabach (bkrabach) added a commit that referenced this pull request Aug 28, 2026
…idempotency section (evidence refuted) (#328)

A payload-level investigation found that three recent changes were built on
a measurement artifact: analyses counted the same events 2-4x across
duplicate snapshot copies of session files (proven by identical
tool_call_id + nanosecond timestamps across "duplicates"). Deduped, the
phenomena each change was built to address vanish. This reverts all three
on that evidence.

1. Remove hooks-dedupe module entirely (PR #323 as hooks-tool-dedupe,
   renamed in PR #326). Deleted modules/hooks-dedupe/ and its `hooks:`
   entry in behaviors/agents.yaml. Deduped, the "same-batch duplicate
   reads" it coalesces do not exist: 0 across 314 sessions / 10,320 reads.
   Measured real value was ~19k tokens (~$0.003) across 5 runs against a
   claimed 16.4% savings -- a ~4,800x overstatement.

   Kept: tests/test_hook_module_classification.py (added by #326). It
   guards a real amplifier-core loader fragility (name-based module-type
   guessing misclassifying a `hooks-*` module as `tool`) that is unrelated
   to whether hooks-dedupe itself exists. Removed only the hooks-dedupe
   specific references: the hardcoded
   `assert "hooks-dedupe" in HOOK_MODULE_IDS` and the
   `test_hooks_dedupe_passes_real_validation` end-to-end test (and its
   now-unused `ModuleValidationError` import). The general, dynamically
   parametrized `test_hook_module_classifies_as_hook` test is untouched
   and still covers every remaining `hooks-*` module.

2. Remove PR #320's "BATCH YOUR DELEGATIONS" guidance block from the
   delegate tool's description in
   modules/tool-delegate/amplifier_module_tool_delegate/__init__.py,
   restoring the original "- Launch multiple agents concurrently when
   tasks are independent" line it replaced. A clean same-commit 5v5 A/B
   measured the guidance as inert (treatment waves median 4 vs control
   median 3) -- the earlier apparent win was a version confound -- while
   it cost ~175 tokens on every single request.

   PR #320's two context-file edits (context/agents/multi-agent-patterns.md,
   context/agents/delegation-instructions.md) are untouched; only the
   scope named above is in this revert. PR #327's own deletions in this
   same string (the CRITICAL/ALWAYS/NEVER preamble and the "DEFAULT TO
   DELEGATION" line) are also untouched -- they stay removed.

3. Remove PR #317's "## Idempotency Discipline" section from
   agents/git-ops.md. Its evidence -- "git-ops re-ran identical clone x8 /
   ls-remote x7 under context truncation" -- is refuted: deduped session
   data shows 8 clones of 8 *different* repos, each cloned once, with
   per-repo ls-remote calls. The thrash it was written to prevent never
   happened.

Verified: full suite green (uv run pytest tests/ -q: 1640 passed, 1
skipped -- down from 1642 by exactly the two hooks-dedupe-specific test
instances removed in (1), both accounted for). python_check clean on all
touched files (only pre-existing, unrelated ruff warnings remain in
tool-delegate's __init__.py, none introduced by this change). Repo-wide
grep confirms modules/hooks-dedupe/ is gone with no dangling functional
references (three remaining mentions of hooks-tool-dedupe/hooks-dedupe are
historical narrative inside the kept regression test's docstring/assert
message, explaining why that general test exists).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-authored-by: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@bkrabach

Copy link
Copy Markdown
Collaborator Author

CORRECTION: this PR's evidentiary basis has been refuted. Deduped payload-level re-analysis shows the "git clone x8 / ls-remote x7 thrash" was 8 DIFFERENT provider repos cloned once each plus per-repo remote checks — legitimate work, not context-truncation thrash. The original count came from duplicate snapshot copies of session event files being counted 2-4x (verified via identical tool_call_id + timestamps). The Idempotency Discipline section is being removed in #328. The mid-PR observation that enable_long_context config is provider-level (not routing-matrix) stands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants