feat(delegate): plan-then-dispatch guidance — batch independent delegations in one turn - #320
Merged
Conversation
…ations in one turn
Measured eval runs showed roots delegating in 3-4 sequential waves, idle
91-94% of wall time, with ~57% of wall structurally recoverable by
batching independent delegations into a single turn. The machinery
already parallelizes multiple delegate calls emitted in one assistant
turn (asyncio.gather, one parallel_group_id per turn) -- the defect is
the root model's one-delegation-wave-per-turn planning policy, not
missing machinery.
This is Stage 1 ("Option A", guidance-only) of the approved
pg1-parallel-delegation spec. Every existing mention of parallelism
argued quality ("different agents bring different tools") but never
argued wall-clock, and never told the model to enumerate its full
delegation set before dispatching. This change replaces the weak/buried
guidance with instructions that state the wall-clock cost directly and
give the model an explicit independence test.
Three verbatim text replacements, no code/schema/behavior changes:
- modules/tool-delegate/amplifier_module_tool_delegate/__init__.py:
replace the delegate tool's "Agent usage notes" block (rendered into
every request's tool spec) with explicit batching guidance.
- context/agents/multi-agent-patterns.md: replace "## Parallel Agent
Dispatch" with "## Plan the Whole Set, Then Dispatch", adding the
wall-clock framing and an explicit independence test.
- context/agents/delegation-instructions.md: replace "## Scaling with
Multiple Instances" with "## Wave Discipline: Batch Everything
Independent", covering both same-agent and cross-agent batching.
Token cost: +175 tokens/request (budgeted in the spec against a ~12,000
token foundation context load). Each edit replaces its section rather
than appending, per ISSUE_HANDLING.md's guidance on not growing files
without cause.
Stage 2 (a `hooks-delegation-batching` nudge hook that reinforces this
guidance at the actual decision point -- after wave N's results land,
before wave N+1 is planned) is fully specified but deferred. It ships
only if Stage 1's DTU measurement misses the >=40% wall-reduction
target.
Verified: full suite green (uv run pytest tests/ -q --tb=short: 1634
passed, 1 skipped), modules/tool-delegate/tests/ green (62 passed),
python_check clean on the touched .py file (only pre-existing warnings
elsewhere in the file, none introduced), and the tool-schema token
estimator's `ast.literal_eval` extraction confirmed still succeeds
(no `return{` substring introduced).
Spec: pg1-parallel-delegation-spec.md (openai_improvement-pg1)
Note for reviewers: after merge, `amplifier reset --remove cache -y` is
required before the new guidance is loaded by a running Amplifier
process -- foundation context files are read from the bundle cache.
🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)
Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Collaborator
Author
|
Merge basis (documented admin-merge — precedent: #317/#318/#319, gemini#39, merged same-day under this pattern)
|
Brian Krabach (bkrabach)
added a commit
that referenced
this pull request
Aug 28, 2026
…idempotency section (evidence refuted) (#328) A payload-level investigation found that three recent changes were built on a measurement artifact: analyses counted the same events 2-4x across duplicate snapshot copies of session files (proven by identical tool_call_id + nanosecond timestamps across "duplicates"). Deduped, the phenomena each change was built to address vanish. This reverts all three on that evidence. 1. Remove hooks-dedupe module entirely (PR #323 as hooks-tool-dedupe, renamed in PR #326). Deleted modules/hooks-dedupe/ and its `hooks:` entry in behaviors/agents.yaml. Deduped, the "same-batch duplicate reads" it coalesces do not exist: 0 across 314 sessions / 10,320 reads. Measured real value was ~19k tokens (~$0.003) across 5 runs against a claimed 16.4% savings -- a ~4,800x overstatement. Kept: tests/test_hook_module_classification.py (added by #326). It guards a real amplifier-core loader fragility (name-based module-type guessing misclassifying a `hooks-*` module as `tool`) that is unrelated to whether hooks-dedupe itself exists. Removed only the hooks-dedupe specific references: the hardcoded `assert "hooks-dedupe" in HOOK_MODULE_IDS` and the `test_hooks_dedupe_passes_real_validation` end-to-end test (and its now-unused `ModuleValidationError` import). The general, dynamically parametrized `test_hook_module_classifies_as_hook` test is untouched and still covers every remaining `hooks-*` module. 2. Remove PR #320's "BATCH YOUR DELEGATIONS" guidance block from the delegate tool's description in modules/tool-delegate/amplifier_module_tool_delegate/__init__.py, restoring the original "- Launch multiple agents concurrently when tasks are independent" line it replaced. A clean same-commit 5v5 A/B measured the guidance as inert (treatment waves median 4 vs control median 3) -- the earlier apparent win was a version confound -- while it cost ~175 tokens on every single request. PR #320's two context-file edits (context/agents/multi-agent-patterns.md, context/agents/delegation-instructions.md) are untouched; only the scope named above is in this revert. PR #327's own deletions in this same string (the CRITICAL/ALWAYS/NEVER preamble and the "DEFAULT TO DELEGATION" line) are also untouched -- they stay removed. 3. Remove PR #317's "## Idempotency Discipline" section from agents/git-ops.md. Its evidence -- "git-ops re-ran identical clone x8 / ls-remote x7 under context truncation" -- is refuted: deduped session data shows 8 clones of 8 *different* repos, each cloned once, with per-repo ls-remote calls. The thrash it was written to prevent never happened. Verified: full suite green (uv run pytest tests/ -q: 1640 passed, 1 skipped -- down from 1642 by exactly the two hooks-dedupe-specific test instances removed in (1), both accounted for). python_check clean on all touched files (only pre-existing, unrelated ruff warnings remain in tool-delegate's __init__.py, none introduced by this change). Repo-wide grep confirms modules/hooks-dedupe/ is gone with no dangling functional references (three remaining mentions of hooks-tool-dedupe/hooks-dedupe are historical narrative inside the kept regression test's docstring/assert message, explaining why that general test exists). 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-authored-by: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Collaborator
Author
|
CORRECTION/OUTCOME: a clean same-foundation-commit 5v5 DTU A/B measured this guidance INERT — treatment waves median 4 vs control 3, wall/token deltas within noise; the earlier apparent front-loading win was a foundation-version confound. The tool-description block is being removed in #328 (~175 tokens/request with no measured benefit). The two context-file sections remain (net-zero replacements). Lesson recorded: prose guidance does not change delegation behavior on these models; mechanical levers do. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stage 1 ("Option A", guidance-only) of the approved
pg1-parallel-delegationspec. Text-only guidance edits — no code, schema, or behavior changes.Measured evidence
A measured eval showed root sessions delegating in 3–4 sequential waves, idle 91–94% of wall time, with ~57% of wall structurally recoverable by batching independent delegations into a single turn. One eval run's wave 2 was
[explorer, explorer, git-ops]— none of which consumed wave 1's output. The root had all the information needed to batch those into wave 1 and did not.Why guidance, not machinery
The parallel-dispatch machinery already works end-to-end and needed no changes:
parallel_group_idper assistant turn and runs every tool call from that turn concurrently viaasyncio.gather(amplifier_module_loop_streaming).parallel_tool_calls=Trueby default; Anthropic never disables it.There is no mechanical serialization to remove. A root that emits three
delegatecalls in one turn already gets three concurrent sub-sessions. The measured idle time is the machinery faithfully executing a one-delegation-per-turn plan — a planning-policy defect, not an execution-machinery defect. PerKERNEL_PHILOSOPHY.md§1, policy belongs at the edges (instructions, tool descriptions), not in new kernel-adjacent code.Every existing mention of parallelism in this repo argued quality ("different agents bring different tools") — none argued wall-clock, and none told the model to enumerate its full delegation set before dispatching. This PR fixes that gap directly at its source.
A companion approach (a
delegations[]array with explicit dependency edges) was evaluated and rejected in the spec: it asks the model for exactly the judgment whose absence is the defect, re-implementsasyncio.gatherone layer below where it already exists correctly, and its marginal wall-clock gain over a correctly-batched guidance fix is under 1%.Changes (3 verbatim text replacements, +175 tokens/request budgeted)
modules/tool-delegate/amplifier_module_tool_delegate/__init__.py— replaced the delegate tool's "Agent usage notes" block (re-rendered into every request's tool spec, last thing read before adelegatecall) with explicit batching guidance and an independence heuristic.context/agents/multi-agent-patterns.md— replaced "## Parallel Agent Dispatch" (framed only as "richer results") with "## Plan the Whole Set, Then Dispatch", adding the wall-clock framing and an explicit independence test.context/agents/delegation-instructions.md— replaced "## Scaling with Multiple Instances" (same-agent only) with "## Wave Discipline: Batch Everything Independent", covering both same-agent and cross-agent batching.Each edit replaces its section rather than appending, per
ISSUE_HANDLING.md's guidance on not growing files without cause. No file grows a section count.Verification
uv run pytest tests/ -q --tb=short(the exact CI command): 1634 passed, 1 skippedmodules/tool-delegate/tests/(62 tests, editable-installed locally to resolve the package): 62 passed — the existing description/schema tests are unaffected by the wording changepython_checkon the touched.pyfile: clean except 8 pre-existing warnings elsewhere in the file (confirmed present before this change viagit stash), none introduced by this editreturn+{substring was introduced —amplifier_foundation.bundle_docs.tool_schema._extract_input_schemastill successfullyast.literal_eval-extracts the tool's static input schema after the edittests/andmodules/*/tests/repo-wide for the replaced phrases ("Launch multiple agents concurrently","Parallel Agent Dispatch","Scaling with Multiple Instances", etc.) — no test asserts the old wording, so no test updates were neededDeferred (not in this PR)
Stage 2 (
hooks-delegation-batching, a nudge hook that reinforces this guidance at the actual decision point — after wave N's results land, before wave N+1 is planned) is fully specified in the spec but not implemented here. Per the spec's rollout plan, it ships only if Stage 1's DTU measurement (median wall-time reduction) misses the ≥40% target. If Stage 1 alone hits target, Stage 2 never ships.B′ (a flat
delegations[]array, no dependency edges) is kept on the shelf as a contingency, gated on a specific signature (a backend whose tool-call parser structurally emits at most one call per turn) that has not been observed. Not scheduled.Post-merge note
After merge,
amplifier reset --remove cache -yis required before the new guidance is loaded by a running Amplifier process — foundation context files are read from the bundle cache, and the guidance is prose read at composition time, not code.🤖 Generated with Amplifier