feat(delegate): flagged sequential-wave nudge (Stage 2 of plan-then-dispatch) - #321
feat(delegate): flagged sequential-wave nudge (Stage 2 of plan-then-dispatch)#321Brian Krabach (bkrabach) wants to merge 1 commit into
Conversation
…ispatch)
Adds `hooks-delegation-batching`, a `tool:post` hook that detects a root
delegating in sequential single-wave turns (a `delegate` call arriving
right after a prior delegation wave completed, with no intervening
non-delegate work) and injects an ephemeral system-reminder nudging it to
batch remaining independent delegations into the current turn.
This is Stage 2 ("Option C") of the plan-then-dispatch parallel-delegation
work. Stage 1 (#320, prose guidance in the delegate tool description and
agent-facing context docs) landed today and widens the first wave, but
does not by itself collapse an already-in-progress sequential habit across
later waves. This hook reinforces the guidance at the actual decision
point -- after wave N's results land, before wave N+1 is planned -- which
is exactly the moment a static context file is furthest from the model's
attention.
Mechanism (proven in-repo): mirrors `hooks-progress-monitor`'s
`HookResult(action="inject_context", ephemeral=True,
append_to_last_tool_result=True)` shape, consumed by the orchestrator
(amplifier-module-loop-streaming, a separate repo) on the next iteration.
Detection: one `parallel_group_id` per assistant turn == one wave. A new
group observed on `tool:post` increments the wave count; a non-delegate
`tool:post` marks the chain "interrupted" and resets it (real work between
waves means the next wave genuinely could not have been planned earlier).
Nudges are deduped per group, capped per session (`max_nudges`), gated on
a minimum wave index (`min_wave_index`), and by default only fire for
narrow (< 2 member) waves (`narrow_wave_only`) since a wide wave already
batched.
Ships opt-in and default-off: `behaviors/delegation-batching.yaml` is not
composed into `behaviors/agents.yaml`, so nobody gets this hook unless
they explicitly add the behavior -- keeping its effect isolated from
Stage 1's guidance for DTU A/B measurement. The module's own `enabled`
config flag (default `true` once composed, matching the pattern in
`behaviors/progress-monitor.yaml`) provides the operational off-switch
within a composed behavior.
Deviations from the spec's copy-verbatim instruction for pyproject.toml
(mechanical, code-layout driven -- the cited exemplar,
hooks-progress-monitor, predates the tests/dev-deps/entry-points
convention used by every other hook module in this repo and has no tests
directory of its own):
- added `[project.entry-points."amplifier.modules"]` (present in
tool-delegate, hooks-process-guard, hooks-deprecation,
hooks-session-naming)
- added `[tool.hatch.build.targets.wheel] packages = [...]` (same modules)
- added `[dependency-groups] dev = ["pytest>=9.0.3",
"pytest-asyncio>=1.3.0"]` and `tests/__init__.py` (matching
hooks-process-guard, the closest analogue with its own test suite) --
otherwise the module's own tests cannot run at all
Tests: `modules/hooks-delegation-batching/tests/test_wave_detection.py`
implements the spec's 10-case table verbatim (single narrow wave, two
consecutive narrow waves, wide-wave dedupe, interruption reset, max-nudges
cap, disabled config, missing/None group_id, internal-exception
swallowing, injection shape, narrow-wave-only suppression) plus additional
coverage for the flag-off no-op path through the real `mount()` wiring,
config/state defaults, and telemetry emission shape. 17/17 pass.
No changes to amplifier-module-loop-streaming (separate repo, out of
scope) or to any existing file in this repo -- Stage 2 is additive only.
After merge: `amplifier reset --remove cache -y` is required before the
new behavior is available to compose (bundle cache).
🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)
Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
|
Closed on A/B evidence per the pre-agreed merge-or-kill gate. 10-run DTU measurement (5 control / 5 treatment, same foundation eb324ad, hook verified mounted in all treatment runs, canary proved the mechanism works when waves are back-to-back): the nudge fired in only 1 of 5 realistic runs — the interrupted-reset means a root doing genuine analysis between delegation waves never accumulates 'consecutive' waves — and in the run where it fired, every subsequent wave was NARROWER, not wider. Median waves 3 (gate: <=2); wall/token deltas were small-sample variance, not treatment effect (mechanism inactive in 4/5 runs). Correct implementation, inert trigger on real workloads. If revived: redesign the trigger on CUMULATIVE narrow-delegation waves regardless of intervening work, and re-measure. Full captures: .amplifier/evaluation/first-turn-ab/20260826-225820/nudge-ab/. |
Summary
Stage 2 ("Option C") of the plan-then-dispatch parallel-delegation work
(spec:
pg1-parallel-delegation-spec.md§7). Addshooks-delegation-batching,a
tool:posthook that detects a root delegating in sequential single-waveturns and injects an ephemeral system-reminder nudging it to batch remaining
independent delegations into the current turn.
Rationale: measured eval runs showed roots delegating in 3–4 sequential
waves, idle 91–94% of wall time. Stage 1 (#320, merged today) widens the
first wave via prose guidance in the delegate tool description and
agent-facing context docs, but doesn't by itself collapse an already-in-progress
sequential habit across later waves. This hook reinforces the same guidance
at the actual decision point — after wave N's results land, before wave N+1
is planned — which is exactly the moment a static context file is furthest
from the model's attention.
Mechanism (proven in-repo, not hypothetical): mirrors
modules/hooks-progress-monitor'sHookResult(action="inject_context", ephemeral=True, append_to_last_tool_result=True)shape. The orchestrator(
amplifier-module-loop-streaming, a separate repo, out of scope here)consumes exactly that shape from
tool:postand applies it on the nextiteration.
Detection: one
parallel_group_idper assistant turn == one wave. A newgroup observed on
tool:postincrements the wave count; a non-delegatetool:postmarks the chain "interrupted" and resets it (real work betweenwaves means the next wave genuinely could not have been planned earlier).
Nudges are deduped per
parallel_group_id, capped per session(
max_nudges, default 3), gated on a minimum wave index (min_wave_index,default 2 — never nudge the first wave), and by default only fire for
narrow (< 2 observed member) waves (
narrow_wave_only, defaulttrue) —a wide wave already batched.
Ships default-off, pending DTU measurement
This ships dark:
behaviors/delegation-batching.yamlis a new, separatebehavior file and is not composed into
behaviors/agents.yamlorbehaviors/tasks.yaml. Nobody's session loads this hook unless theyexplicitly add the behavior — this keeps its effect isolable from Stage 1's
guidance for the DTU A/B measurement described in the spec (§8.3, §9): only
if Stage 1 alone misses the wall-clock-reduction target does this Stage 2
behavior get composed into a default bundle (§9, stage "2b").
The module's own
enabledconfig flag (defaulttrueonce the behavior isexplicitly composed — matching the existing
behaviors/progress-monitor.yamlpattern) is the operational off-switch within a composed behavior, per the
spec's rollback note (§9): "Stage 2 is
enabled: falseor dropping thebehavior from composition — it is opt-in, so no rollback is needed for
anyone who did not opt in."
Deviation from the spec (mechanical, code-layout driven)
The spec's §7.2 instructs copying
modules/hooks-progress-monitor/pyproject.tomlverbatim (name/description only). That exemplar predates the
tests/dev-deps/entry-points convention used by every other hook module in
this repo (
hooks-process-guard,hooks-deprecation,hooks-session-naming,tool-delegate) and has no tests directory of its own — copying it verbatimwould leave this module's own test suite unable to run at all (no
pytestdev dependency). Added, mirroring the closest analogue
(
hooks-process-guard):[project.entry-points."amplifier.modules"][tool.hatch.build.targets.wheel] packages = [...][dependency-groups] dev = ["pytest>=9.0.3", "pytest-asyncio>=1.3.0"]tests/__init__.pyNo other deviation from the spec's algorithm (§7.3), injection text (§7.4,
verbatim), or telemetry contract (§7.6).
What changed
modules/hooks-delegation-batching/pyproject.tomlmodules/hooks-delegation-batching/README.mdmodules/hooks-delegation-batching/amplifier_module_hooks_delegation_batching/__init__.pyDelegationBatchingConfig,DelegationBatchingState,DelegationBatchingHooks,mount()modules/hooks-delegation-batching/tests/test_wave_detection.pymodules/hooks-delegation-batching/tests/__init__.pymodules/hooks-delegation-batching/uv.lockbehaviors/delegation-batching.yamlagents.yamlNo existing file is modified. No change to
amplifier-module-loop-streaming(separate repo) or any provider module, per the spec's scope (§3).
Testing
cd modules/hooks-delegation-batching && uv run pytest tests/ -v— 17/17 passed(spec's 10-case table in full, plus flag-off-via-
mount(), config/statedefaults, and telemetry-emission-shape coverage)
python_checkclean on touched files (module-scopedBLE001blind-exceptand the environment-only
pyright reportMissingImportsare bothpre-existing baseline noise — identical profile on the known-good
hooks-process-guardmodule checked for comparison)uv run pytest tests/ -q(repo root) — 1635 passed, 1 skipped (noregressions)
Post-merge
amplifier reset --remove cache -yis required before this behavior isavailable to compose — the foundation context/bundle files are read from
the bundle cache, and a running process has the old ones.
Not merging
Per the task: this PR ships default-off pending DTU A/B measurement (spec
§8.3, §9). Do not merge without that measurement.
🤖 Generated with Amplifier