Skip to content

feat(delegate): flagged sequential-wave nudge (Stage 2 of plan-then-dispatch) - #321

Closed
Brian Krabach (bkrabach) wants to merge 1 commit into
mainfrom
feat/delegation-wave-nudge
Closed

feat(delegate): flagged sequential-wave nudge (Stage 2 of plan-then-dispatch)#321
Brian Krabach (bkrabach) wants to merge 1 commit into
mainfrom
feat/delegation-wave-nudge

Conversation

@bkrabach

Copy link
Copy Markdown
Collaborator

Summary

Stage 2 ("Option C") of the plan-then-dispatch parallel-delegation work
(spec: pg1-parallel-delegation-spec.md §7). Adds hooks-delegation-batching,
a tool:post hook that detects a root delegating in sequential single-wave
turns and injects an ephemeral system-reminder nudging it to batch remaining
independent delegations into the current turn.

Rationale: measured eval runs showed roots delegating in 3–4 sequential
waves, idle 91–94% of wall time. Stage 1 (#320, merged today) widens the
first wave via prose guidance in the delegate tool description and
agent-facing context docs, but doesn't by itself collapse an already-in-progress
sequential habit across later waves. This hook reinforces the same guidance
at the actual decision point — after wave N's results land, before wave N+1
is planned — which is exactly the moment a static context file is furthest
from the model's attention.

Mechanism (proven in-repo, not hypothetical): mirrors
modules/hooks-progress-monitor's
HookResult(action="inject_context", ephemeral=True, append_to_last_tool_result=True) shape. The orchestrator
(amplifier-module-loop-streaming, a separate repo, out of scope here)
consumes exactly that shape from tool:post and applies it on the next
iteration.

Detection: one parallel_group_id per assistant turn == one wave. A new
group observed on tool:post increments the wave count; a non-delegate
tool:post marks the chain "interrupted" and resets it (real work between
waves means the next wave genuinely could not have been planned earlier).
Nudges are deduped per parallel_group_id, capped per session
(max_nudges, default 3), gated on a minimum wave index (min_wave_index,
default 2 — never nudge the first wave), and by default only fire for
narrow (< 2 observed member) waves (narrow_wave_only, default true) —
a wide wave already batched.

Ships default-off, pending DTU measurement

This ships dark: behaviors/delegation-batching.yaml is a new, separate
behavior file and is not composed into behaviors/agents.yaml or
behaviors/tasks.yaml. Nobody's session loads this hook unless they
explicitly add the behavior — this keeps its effect isolable from Stage 1's
guidance for the DTU A/B measurement described in the spec (§8.3, §9): only
if Stage 1 alone misses the wall-clock-reduction target does this Stage 2
behavior get composed into a default bundle (§9, stage "2b").

The module's own enabled config flag (default true once the behavior is
explicitly composed — matching the existing behaviors/progress-monitor.yaml
pattern) is the operational off-switch within a composed behavior, per the
spec's rollback note (§9): "Stage 2 is enabled: false or dropping the
behavior from composition — it is opt-in, so no rollback is needed for
anyone who did not opt in."

Deviation from the spec (mechanical, code-layout driven)

The spec's §7.2 instructs copying modules/hooks-progress-monitor/pyproject.toml
verbatim (name/description only). That exemplar predates the
tests/dev-deps/entry-points convention used by every other hook module in
this repo (hooks-process-guard, hooks-deprecation, hooks-session-naming,
tool-delegate) and has no tests directory of its own — copying it verbatim
would leave this module's own test suite unable to run at all (no pytest
dev dependency). Added, mirroring the closest analogue
(hooks-process-guard):

  • [project.entry-points."amplifier.modules"]
  • [tool.hatch.build.targets.wheel] packages = [...]
  • [dependency-groups] dev = ["pytest>=9.0.3", "pytest-asyncio>=1.3.0"]
  • tests/__init__.py

No other deviation from the spec's algorithm (§7.3), injection text (§7.4,
verbatim), or telemetry contract (§7.6).

What changed

File Change
modules/hooks-delegation-batching/pyproject.toml new
modules/hooks-delegation-batching/README.md new
modules/hooks-delegation-batching/amplifier_module_hooks_delegation_batching/__init__.py new — DelegationBatchingConfig, DelegationBatchingState, DelegationBatchingHooks, mount()
modules/hooks-delegation-batching/tests/test_wave_detection.py new — 10-case spec table + flag-off/mount/telemetry coverage
modules/hooks-delegation-batching/tests/__init__.py new
modules/hooks-delegation-batching/uv.lock new
behaviors/delegation-batching.yaml new — opt-in, not composed into agents.yaml

No existing file is modified. No change to amplifier-module-loop-streaming
(separate repo) or any provider module, per the spec's scope (§3).

Testing

  • cd modules/hooks-delegation-batching && uv run pytest tests/ -v17/17 passed
    (spec's 10-case table in full, plus flag-off-via-mount(), config/state
    defaults, and telemetry-emission-shape coverage)
  • python_check clean on touched files (module-scoped BLE001 blind-except
    and the environment-only pyright reportMissingImports are both
    pre-existing baseline noise — identical profile on the known-good
    hooks-process-guard module checked for comparison)
  • uv run pytest tests/ -q (repo root) — 1635 passed, 1 skipped (no
    regressions)

Post-merge

amplifier reset --remove cache -y is required before this behavior is
available to compose — the foundation context/bundle files are read from
the bundle cache, and a running process has the old ones.

Not merging

Per the task: this PR ships default-off pending DTU A/B measurement (spec
§8.3, §9). Do not merge without that measurement.

🤖 Generated with Amplifier

…ispatch)

Adds `hooks-delegation-batching`, a `tool:post` hook that detects a root
delegating in sequential single-wave turns (a `delegate` call arriving
right after a prior delegation wave completed, with no intervening
non-delegate work) and injects an ephemeral system-reminder nudging it to
batch remaining independent delegations into the current turn.

This is Stage 2 ("Option C") of the plan-then-dispatch parallel-delegation
work. Stage 1 (#320, prose guidance in the delegate tool description and
agent-facing context docs) landed today and widens the first wave, but
does not by itself collapse an already-in-progress sequential habit across
later waves. This hook reinforces the guidance at the actual decision
point -- after wave N's results land, before wave N+1 is planned -- which
is exactly the moment a static context file is furthest from the model's
attention.

Mechanism (proven in-repo): mirrors `hooks-progress-monitor`'s
`HookResult(action="inject_context", ephemeral=True,
append_to_last_tool_result=True)` shape, consumed by the orchestrator
(amplifier-module-loop-streaming, a separate repo) on the next iteration.

Detection: one `parallel_group_id` per assistant turn == one wave. A new
group observed on `tool:post` increments the wave count; a non-delegate
`tool:post` marks the chain "interrupted" and resets it (real work between
waves means the next wave genuinely could not have been planned earlier).
Nudges are deduped per group, capped per session (`max_nudges`), gated on
a minimum wave index (`min_wave_index`), and by default only fire for
narrow (< 2 member) waves (`narrow_wave_only`) since a wide wave already
batched.

Ships opt-in and default-off: `behaviors/delegation-batching.yaml` is not
composed into `behaviors/agents.yaml`, so nobody gets this hook unless
they explicitly add the behavior -- keeping its effect isolated from
Stage 1's guidance for DTU A/B measurement. The module's own `enabled`
config flag (default `true` once composed, matching the pattern in
`behaviors/progress-monitor.yaml`) provides the operational off-switch
within a composed behavior.

Deviations from the spec's copy-verbatim instruction for pyproject.toml
(mechanical, code-layout driven -- the cited exemplar,
hooks-progress-monitor, predates the tests/dev-deps/entry-points
convention used by every other hook module in this repo and has no tests
directory of its own):
- added `[project.entry-points."amplifier.modules"]` (present in
  tool-delegate, hooks-process-guard, hooks-deprecation,
  hooks-session-naming)
- added `[tool.hatch.build.targets.wheel] packages = [...]` (same modules)
- added `[dependency-groups] dev = ["pytest>=9.0.3",
  "pytest-asyncio>=1.3.0"]` and `tests/__init__.py` (matching
  hooks-process-guard, the closest analogue with its own test suite) --
  otherwise the module's own tests cannot run at all

Tests: `modules/hooks-delegation-batching/tests/test_wave_detection.py`
implements the spec's 10-case table verbatim (single narrow wave, two
consecutive narrow waves, wide-wave dedupe, interruption reset, max-nudges
cap, disabled config, missing/None group_id, internal-exception
swallowing, injection shape, narrow-wave-only suppression) plus additional
coverage for the flag-off no-op path through the real `mount()` wiring,
config/state defaults, and telemetry emission shape. 17/17 pass.

No changes to amplifier-module-loop-streaming (separate repo, out of
scope) or to any existing file in this repo -- Stage 2 is additive only.

After merge: `amplifier reset --remove cache -y` is required before the
new behavior is available to compose (bundle cache).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@bkrabach

Copy link
Copy Markdown
Collaborator Author

Closed on A/B evidence per the pre-agreed merge-or-kill gate. 10-run DTU measurement (5 control / 5 treatment, same foundation eb324ad, hook verified mounted in all treatment runs, canary proved the mechanism works when waves are back-to-back): the nudge fired in only 1 of 5 realistic runs — the interrupted-reset means a root doing genuine analysis between delegation waves never accumulates 'consecutive' waves — and in the run where it fired, every subsequent wave was NARROWER, not wider. Median waves 3 (gate: <=2); wall/token deltas were small-sample variance, not treatment effect (mechanism inactive in 4/5 runs). Correct implementation, inert trigger on real workloads. If revived: redesign the trigger on CUMULATIVE narrow-delegation waves regardless of intervening work, and re-measure. Full captures: .amplifier/evaluation/first-turn-ab/20260826-225820/nudge-ab/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants