From 30508604bc4824e4b6aa390c5ca5c505f860badd Mon Sep 17 00:00:00 2001 From: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com> Date: Thu, 27 Aug 2026 07:21:14 -0700 Subject: [PATCH] feat(delegate): flag-gated per-leg call budget via orchestrator max_iterations (Layer 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Layered Bounding for Delegated Sessions (spec: 298-replacement, replacing the wall-clock-only default in #298). Adds a per-session-leg LLM-call budget as the first line of defense in front of the delegate's existing settings.timeout wall-clock backstop, delivered with zero new kernel surface: tool-delegate writes max_iterations (and a new budget_warn_ratio) into the orchestrator_config dict it already passes to spawn_fn, and amplifier-app-cli's session_spawner already does a caller-wins .update() into the child's config -- zero app-cli changes needed. Enforcement itself lives in the orchestrator loop (see companion PR microsoft/amplifier-module-loop-streaming#43), which already counts LLM calls via max_iterations and already exits exhaustion via a normal return (graceful wrap-up), so the resulting transcript is complete and resumable -- unlike a cancellation-based timeout. Ships DARK: settings.max_llm_calls defaults to None, so no budget is injected into any child session and orchestrator_config is byte-for-byte what it was before this change. Nothing here changes behavior until an operator explicitly sets settings.max_llm_calls. Precedence chain (highest first): 1. Per-call tool input (`max_llm_calls`) -- implemented 2. Per-agent frontmatter (`agents[name]["budget"]["max_llm_calls"]`) -- NOT implemented, see below 3. This module's settings.max_llm_calls (default None) -- implemented 4. Inherited parent orchestrator_config's max_iterations -- implemented (the pre-existing inheritance path, left untouched when no budget applies) Per-agent frontmatter override (rank 2) does not ship: verified empirically (not just read from source) that a top-level `budget:` block in an agent .md's frontmatter is dropped by amplifier_foundation.bundle._dataclass._load_agent_file_metadata, which only forwards a fixed allowlist of top-level keys (tools, providers, hooks, session, provider_preferences, model_role, agents) -- budget is not among them. Reproduced in tests/test_delegate_call_budget.py::test_agent_frontmatter_budget_key_is_dropped. Ranks 1, 3, and 4 ship; rank 2 is a follow-up requiring a change to the frontmatter loader itself, documented in this module's README "Known gaps" section. Also adds: - Eager validation (_validate_call_budget / _check_call_budget_type): reject bool, non-int, and negative values at the point supplied (module construction for the settings default, execute() for the per-call override) -- never at spawn time. - Negotiated-feature warning (spec §4.4): if a budget was requested but the child's orchestrator reports no llm_call_budget telemetry (e.g. a third-party orchestrator with no max_iterations support), logs a warning and sets metadata.budget_enforced = false on the returned ToolResult, so the gap is loud rather than silent. - max_llm_calls entry in the tool's input schema (kept a pure literal for the static token-cost estimator). Files: - modules/tool-delegate/amplifier_module_tool_delegate/__init__.py: _check_call_budget_type / _validate_call_budget module functions; settings.max_llm_calls / budget_warn_ratio in __init__; per-call max_llm_calls parsing + validation in execute(); _resolve_call_budget method; orchestrator_config build (copy-not-mutate + budget injection) and negotiated-feature warning in _spawn_new_session; max_llm_calls schema entry - modules/tool-delegate/README.md: "Layer 1 call budget" section + "Known gaps" - modules/tool-delegate/tests/test_delegate_call_budget.py (new): T2.1, T2.2, T2.4, T2.5, T2.6, T2.7, T2.8, T2.9, T2.10, T2.11 + the frontmatter round-trip verification test (14 tests) Testing: - New tests: 14 passed - Full tool-delegate module suite: 80 passed (was ~66; zero regressions) - Full foundation repo suite (tests/): 1634 passed, 1 skipped -- matches pre-change baseline exactly - ruff/pyright: no new issues vs baseline Part of the 298-replacement design (Layer 1 of 3). Companion PR: microsoft/amplifier-module-loop-streaming#43 (orchestrator-side enforcement). #298 is being revised separately to reframe its wall-clock default as the Layer 3 backstop behind this budget. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com> --- modules/tool-delegate/README.md | 79 ++++ .../__init__.py | 188 +++++++- .../tests/test_delegate_call_budget.py | 445 ++++++++++++++++++ 3 files changed, 708 insertions(+), 4 deletions(-) create mode 100644 modules/tool-delegate/tests/test_delegate_call_budget.py diff --git a/modules/tool-delegate/README.md b/modules/tool-delegate/README.md index dd29d1d..11ff3ec 100644 --- a/modules/tool-delegate/README.md +++ b/modules/tool-delegate/README.md @@ -45,6 +45,7 @@ Agent's explicit tool declarations are always honored, even when parent excludes | `context_turns` | integer | 5 | Number of turns when context_depth is 'recent' | | `context_scope` | enum | "conversation" | Which content: conversation, agents, full | | `provider_preferences` | array | - | Ordered provider/model preferences | +| `max_llm_calls` | integer | - | Override the Layer 1 LLM-call budget for this delegation (per session leg). `0` disables the budget for this call. Only takes effect when `settings.max_llm_calls` is configured -- see "Layer 1 call budget" below. | ## Configuration @@ -70,6 +71,16 @@ modules: - delegate # Default: spawned agents can't further delegate exclude_hooks: [] timeout: 300 + max_llm_calls: null # Layer 1 call budget (spec: 298-replacement). + # None/unset (default): ships dark -- no + # budget is injected into any child session; + # today's behavior is unchanged. Set to a + # positive integer to enforce a per-leg + # LLM-call budget (see below). 0 is + # equivalent to null (explicit no-budget). + budget_warn_ratio: 0.8 # Fraction of max_llm_calls at which a + # one-shot "start converging" warning fires. + # Only meaningful when max_llm_calls is set. ``` ### Structured Return Contract @@ -136,6 +147,74 @@ opt-in matrix. `not_covered_count`, and `artifacts_count` (all `int | None`, `None` alongside `contract_conformant is None`). +## Layer 1 call budget + +This module can bound how many main-loop LLM calls a delegated child +session may make in one leg, as a first line of defense in front of +`settings.timeout`'s wall-clock backstop (see spec: 298-replacement, +"Layered Bounding for Delegated Sessions"). The enforcement mechanism is +the child's own orchestrator `max_iterations` config (e.g. +`amplifier-module-loop-streaming`'s streaming loop) -- this module does not +count LLM calls itself; it only injects a value into the child's +`orchestrator_config` at spawn time. + +**Ships dark.** `settings.max_llm_calls` defaults to `None`, which means no +budget is injected at all -- every child session gets exactly the +`orchestrator_config` its parent would have given it anyway (rank 4 below). +Nothing about this feature is active until an operator sets +`settings.max_llm_calls` to a positive integer. + +### Precedence (highest first) + +| Rank | Source | Key | Status | +|------|--------|-----|--------| +| 1 | Per-call tool input | `max_llm_calls` | Implemented | +| 2 | Per-agent frontmatter | `agents[agent_name]["budget"]["max_llm_calls"]` | **Not implemented -- see "Known gaps" below** | +| 3 | This module's setting | `settings.max_llm_calls` | Implemented (default `null`) | +| 4 | Inherited parent `orchestrator_config` | `max_iterations` (only if 1 and 3 are both absent) | Implemented (pre-existing inheritance path, untouched) | + +An explicit `0` at rank 1 means "no Layer 1 budget for this delegation" -- +the wall-clock backstop (`settings.timeout`) still applies. Negative values +and booleans are rejected at the point they are supplied (fail loud, not a +silent coercion). + +### Negotiated feature, not a contract requirement + +`max_iterations` is an **advisory** orchestrator config convention (see +`amplifier-core/docs/contracts/ORCHESTRATOR_CONTRACT.md`), not a required +one. If a budget was requested but the child's orchestrator doesn't +implement `max_iterations` at all (a third-party orchestrator, or one with +no budget support), the child's `orchestrator:complete` metadata will carry +no `llm_call_budget` key. This module detects that and: + +- logs a warning naming the agent and the fact that only the wall-clock + backstop applies +- sets `metadata.budget_enforced: false` on the returned `ToolResult` + +so the gap is loud, never silent. + +### Known gaps + +- **Per-agent frontmatter override (precedence rank 2) is not + implemented.** A top-level `budget:` block in an agent `.md` file's + frontmatter does **not** currently survive into + `coordinator.config["agents"][name]`: + `amplifier_foundation.bundle._dataclass._load_agent_file_metadata` only + forwards a fixed allowlist of top-level frontmatter keys (`tools`, + `providers`, `hooks`, `session`, `provider_preferences`, `model_role`, + `agents`) -- `budget` is not among them, and is silently dropped. This was + verified empirically (not just read from the source) -- see + `tests/test_delegate_call_budget.py`'s + `test_agent_frontmatter_budget_key_is_dropped`. Ranks 1, 3, and 4 of the + precedence chain ship and work today; rank 2 is a follow-up that requires + a change in `amplifier-foundation`'s agent-frontmatter loader, not this + module. +- **Cross-provider healthy-`llm_calls` distribution is not yet measured.** + Any default set for `settings.max_llm_calls` today is a hypothesis, not a + measurement -- see the spec's staged-rollout plan (S0 telemetry-only -> S1 + warn-only -> S2 generous enforcement -> S3 target enforcement) before + setting a production default. + ## Note This module is recommended over `tool-task` for new development due to its enhanced context control and bug fixes. diff --git a/modules/tool-delegate/amplifier_module_tool_delegate/__init__.py b/modules/tool-delegate/amplifier_module_tool_delegate/__init__.py index 241d445..9ba0cda 100644 --- a/modules/tool-delegate/amplifier_module_tool_delegate/__init__.py +++ b/modules/tool-delegate/amplifier_module_tool_delegate/__init__.py @@ -183,6 +183,51 @@ def _return_contract_event_fields(contract: dict[str, Any]) -> dict[str, Any]: ) +def _check_call_budget_type(value: int) -> None: + """Raise if ``value`` is not a valid LLM-call budget integer. + + Rejects ``bool`` (a ``bool`` is an ``int`` subclass in Python; + ``True``/``False`` must never silently become ``1``/``0`` here), any + other non-``int``, and negative values. Mirrors the validation + discipline established for ``settings.timeout`` in the #298 branch + (``_validate_timeout``): fail loud at the point the value is supplied, + never at spawn time. + + Deliberately does NOT collapse ``0`` -- callers that must distinguish + "explicitly zero" from "not supplied" (e.g. the per-call precedence + rank in ``_resolve_call_budget``) need the raw value. Callers for whom + the two are equivalent should use ``_validate_call_budget`` instead. + """ + if isinstance(value, bool): + raise TypeError(f"max_llm_calls must be an integer, not a bool: {value!r}") + if not isinstance(value, int): + raise TypeError( + "max_llm_calls must be an integer or None, got " + f"{type(value).__name__}: {value!r}" + ) + if value < 0: + raise ValueError(f"max_llm_calls must be >= 0, got {value}") + + +def _validate_call_budget(value: Any) -> int | None: + """Validate + collapse a Layer 1 LLM-call budget value. + + ``None`` and ``0`` both mean "no Layer 1 budget" -- collapsed to + ``None`` so this caller only needs one falsy check. Use this for + sources where "unset" and "explicitly zero" are equivalent (this + module's own ``settings.max_llm_calls`` default). For the per-call + override, where an explicit ``0`` must override a non-zero default + rather than being indistinguishable from "not supplied", use + ``_check_call_budget_type`` directly and let + ``DelegateTool._resolve_call_budget`` do the collapse at the point it + knows the value was explicitly given. + """ + if value is None: + return None + _check_call_budget_type(value) + return value or None # 0 -> None (explicit opt-out) + + async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = None): """Mount the agent delegation tool. @@ -289,6 +334,20 @@ def __init__(self, coordinator: ModuleCoordinator, config: dict[str, Any]): # "delegate:model_role_unresolved" event is emitted either way. self.strict_model_role: bool = settings.get("strict_model_role", False) + # Layer 1 (per-leg LLM-call budget, spec: 298-replacement) settings. + # Ships DARK at S0: default None means "inject no budget at all" -- + # today's behavior (whatever max_iterations the parent's own + # orchestrator config already carries, typically unlimited -- see + # _spawn_new_session's orchestrator_config build) is completely + # unchanged until settings.max_llm_calls is explicitly set to a + # positive integer. See _resolve_call_budget for the precedence + # chain, and the module README's "Known gaps" section for why a + # per-agent frontmatter override (spec §6.1) is NOT implemented. + self.max_llm_calls: int | None = _validate_call_budget( + settings.get("max_llm_calls") + ) + self.budget_warn_ratio: float = float(settings.get("budget_warn_ratio", 0.8)) + # Build feature registry for dynamic description composition self._feature_registry = self._build_feature_registry() @@ -617,6 +676,16 @@ def _static_input_schema(self) -> dict: "Available roles are shown in the session context." ), }, + "max_llm_calls": { + "type": "integer", + "description": ( + "Override the LLM-call budget for this delegation (per session " + "leg). Raise for known-large tasks; 0 disables the budget for " + "this call (a wall-clock backstop still applies). Only takes " + "effect when this session's settings.max_llm_calls is " + "configured -- most deployments do not set one yet." + ), + }, }, "required": ["instruction"], } @@ -1262,6 +1331,30 @@ async def execute(self, input: dict) -> ToolResult: "default model." ) + # Layer 1 call-budget: per-call override (spec: 298-replacement, + # highest-precedence rank). None means "no override supplied" -- + # falls through to this module's own settings.max_llm_calls default + # in _resolve_call_budget. Validated eagerly here (not deferred to + # spawn) so a bad value is reported against the call that supplied + # it, matching the "Validate instruction" check just below. + # + # Deliberately NOT collapsed via _validate_call_budget: an explicit + # 0 here must be distinguishable from "not supplied" (None), so + # _resolve_call_budget can tell "opt out of the budget for this one + # call" apart from "say nothing, use the module default" -- the + # collapse (0 -> None) happens there, once that distinction has + # already been used. + raw_max_llm_calls = input.get("max_llm_calls") + call_budget_override: int | None = None + if raw_max_llm_calls is not None: + try: + _check_call_budget_type(raw_max_llm_calls) + except (TypeError, ValueError) as e: + return ToolResult( + success=False, error={"message": f"Invalid max_llm_calls: {e}"} + ) + call_budget_override = raw_max_llm_calls + # Validate instruction (always required) if not instruction: return ToolResult( @@ -1355,8 +1448,46 @@ async def execute(self, input: dict) -> ToolResult: parallel_group_id=parallel_group_id, raw_model_role=raw_model_role, agents=agents, + call_budget_override=call_budget_override, ) + def _resolve_call_budget( + self, + agent_name: str, + call_override: int | None, + ) -> int | None: + """Resolve the per-leg LLM-call budget for a delegation. + + ``None`` means "no Layer 1 budget for this delegation" -- Layer 3 + (the delegate's own wall-clock ``settings.timeout``) still applies + regardless. + + Precedence (highest first): + 1. ``call_override`` -- the per-call ``max_llm_calls`` tool input, + already validated by ``execute()``. + 2. ``self.max_llm_calls`` -- this module's ``settings.max_llm_calls`` + default (``None`` at S0 -- ships dark). + + NOT implemented: a per-agent frontmatter override + (``agents[agent_name]["budget"]["max_llm_calls"]``, spec §6.1, + precedence rank 2 of 4). Verified empirically that a top-level + ``budget:`` block in an agent ``.md``'s frontmatter is dropped -- + ``amplifier_foundation.bundle._dataclass._load_agent_file_metadata`` + only forwards a fixed allowlist of top-level keys (``tools``, + ``providers``, ``hooks``, ``session``, ``provider_preferences``, + ``model_role``, ``agents``); ``budget`` is not among them. Wiring + this rank now would silently no-op for every agent file. See + ``tests/test_delegate_call_budget.py``'s + ``test_agent_frontmatter_budget_key_is_dropped`` for the + reproducing test, and the module README's "Known gaps" section. + ``agent_name`` is accepted (and intentionally unused today) so this + signature does not need to change again once that gap is closed. + """ + del agent_name # unused until per-agent frontmatter budget lands + if call_override is not None: + return call_override or None # 0 -> None (explicit opt-out) + return self.max_llm_calls + async def _spawn_new_session( self, agent_name: str, @@ -1371,6 +1502,7 @@ async def _spawn_new_session( parallel_group_id: str | None = None, raw_model_role: str = "", agents: dict | None = None, + call_budget_override: int | None = None, ) -> ToolResult: """Spawn a new agent sub-session. @@ -1385,6 +1517,9 @@ async def _spawn_new_session( context_turns: Number of recent turns (when context_depth="recent") provider_preferences: Resolved provider preferences list hooks: Hook coordinator for event emission + call_budget_override: Per-call Layer 1 budget override (spec: + 298-replacement), already validated by execute(). None means + "no override" -- falls through to settings.max_llm_calls. tool_call_id: Orchestrator tool call ID (enriches event payloads) parallel_group_id: Parallel group ID (enriches event payloads) raw_model_role: Raw model role string for routing tracking @@ -1499,17 +1634,31 @@ async def _spawn_new_session( # Extract orchestrator config from parent session for inheritance. # Guard with isinstance to handle non-dict orchestrator values gracefully # (e.g. when orchestrator is a string like "loop-basic"). - orchestrator_config = None + orchestrator_config: dict[str, Any] = {} parent_config = parent_session.config or {} session_config = parent_config.get("session", {}) orch_section = session_config.get("orchestrator", {}) if isinstance(orch_section, dict): if orch_config := orch_section.get("config"): - orchestrator_config = orch_config + # Copy: never mutate the parent's own config dict below. + orchestrator_config = dict(orch_config) logger.debug( f"Inheriting orchestrator config: {orchestrator_config}" ) + # Layer 1: resolve the per-leg LLM-call budget for this child + # (spec: 298-replacement). None means "no Layer 1 budget" -- + # the key is then left untouched, so whatever max_iterations + # the parent's own inherited orchestrator config already + # carries (rank 4 -- typically unlimited) is what the child + # gets. Ships dark at S0: settings.max_llm_calls defaults to + # None, so this is a no-op until a caller opts in. + call_budget = self._resolve_call_budget(agent_name, call_budget_override) + if call_budget is not None: + orchestrator_config["max_iterations"] = call_budget + orchestrator_config["budget_warn_ratio"] = self.budget_warn_ratio + orchestrator_config_out: dict[str, Any] | None = orchestrator_config or None + # Calculate self-delegation depth for child session # Named agents reset to 0, self-delegation increments if agent_name == "self": @@ -1557,7 +1706,7 @@ async def _spawn_new_session( sub_session_id=sub_session_id, tool_inheritance=tool_inheritance, hook_inheritance=hook_inheritance, - orchestrator_config=orchestrator_config, + orchestrator_config=orchestrator_config_out, provider_preferences=provider_preferences, self_delegation_depth=child_self_delegation_depth, session_metadata=session_metadata, @@ -1591,6 +1740,31 @@ async def _spawn_new_session( }, ) + # Negotiated-feature seam (spec: 298-replacement §4.4). A budget + # was requested (call_budget is not None) but the child's + # orchestrator reported no llm_call_budget telemetry -- either + # it doesn't implement max_iterations at all (e.g. a + # third-party orchestrator, or loop-basic), or it silently + # ignored the config key. Layer 1 bounding is NOT active for + # this delegation in that case; only the wall-clock backstop + # (settings.timeout) applies. Silence is the failure mode this + # spec exists to eliminate, so make it loud rather than let the + # caller believe a budget is enforced when it is not. + result_metadata = result.get("metadata") or {} + budget_enforced = True + if call_budget is not None: + budget_enforced = "llm_call_budget" in result_metadata + if not budget_enforced: + logger.warning( + "Delegate requested an LLM-call budget of %s for agent " + "%r, but the child's orchestrator reported no budget " + "telemetry. Layer 1 bounding is NOT active for this " + "delegation; only the %ss wall-clock backstop applies.", + call_budget, + agent_name, + self.timeout, + ) + # Build provider routing summary (only when routing was requested) # Always include both keys for a stable dict shape — consumers # can safely read provider_routing["model_role"] without KeyError. @@ -1605,6 +1779,12 @@ async def _spawn_new_session( ), } + # Merge the budget_enforced flag into the metadata bag we + # forward, without mutating the child's own returned dict. + output_metadata = dict(result_metadata) + if call_budget is not None: + output_metadata["budget_enforced"] = budget_enforced + # Return output with session_id for multi-turn capability. # "response" is `cleaned_response` -- byte-identical to # result["output"] whenever the feature is disabled, parsing @@ -1620,7 +1800,7 @@ async def _spawn_new_session( "agent": agent_name, "turn_count": result.get("turn_count", 1), "status": result.get("status", "success"), - "metadata": result.get("metadata", {}), + "metadata": output_metadata, "contract": contract, **( {"provider_routing": provider_routing} diff --git a/modules/tool-delegate/tests/test_delegate_call_budget.py b/modules/tool-delegate/tests/test_delegate_call_budget.py new file mode 100644 index 0000000..e7285fa --- /dev/null +++ b/modules/tool-delegate/tests/test_delegate_call_budget.py @@ -0,0 +1,445 @@ +"""Tests for the Layer 1 LLM-call budget (spec: 298-replacement, §3, §11 T2). + +Covers, per the implementation spec's test plan (T2 series): + T2.1 Default injection -- settings.max_llm_calls flows into + orchestrator_config["max_iterations"] + T2.2 Per-call override -- input["max_llm_calls"] wins + T2.4 Precedence -- per-call beats the module setting + T2.5 Opt-out -- max_llm_calls: 0 -> key absent from orchestrator_config + T2.6 Parent config preserved -- other keys survive, parent dict untouched + T2.7 Validation -- bad values raise at construction, not at spawn + T2.8 Status passthrough -- budget_exhausted status forwarded verbatim + T2.9 Metadata passthrough -- forwarded, plus budget_enforced when relevant + T2.10 Negotiated-feature warning -- missing llm_call_budget telemetry + T2.11 Resume carries no orchestrator_config + +Plus the frontmatter round-trip verification the spec's open item #1 +requires before shipping a per-agent override (it does not round-trip -- +see test_agent_frontmatter_budget_key_is_dropped below), and a "ships +dark" regression proving default behavior is unchanged at S0. +""" + +from __future__ import annotations + +import tempfile +from pathlib import Path +from unittest.mock import AsyncMock, MagicMock + +import pytest +from amplifier_module_tool_delegate import DelegateTool, _validate_call_budget + +# --------------------------------------------------------------------------- +# Helpers (pattern mirrors tests/test_delegate_spawn_new_session.py) +# --------------------------------------------------------------------------- + + +def _make_tool( + *, + settings: dict | None = None, + orchestrator_value: dict | str | None = None, + spawn_result: dict | None = None, +) -> tuple[DelegateTool, AsyncMock]: + """Create a DelegateTool wired for _spawn_new_session()-level tests. + + Returns (tool, spawn_fn) so tests can inspect spawn_fn.call_args. + """ + result = spawn_result or { + "output": "done", + "session_id": "child-001", + "status": "success", + "turn_count": 1, + "metadata": {}, + } + spawn_fn = AsyncMock(return_value=result) + + coordinator = MagicMock() + coordinator.session_id = "parent-session-123" + coordinator.config = {"agents": {}} + coordinator.session_state = {} + coordinator._tool_dispatch_context = {} + coordinator.get_capability = lambda name: ( + spawn_fn if name == "session.spawn" else None + ) + coordinator.get = MagicMock(return_value=None) + + parent_session = MagicMock() + parent_session.session_id = "parent-session-123" + parent_session.config = { + "session": { + "orchestrator": ( + orchestrator_value if orchestrator_value is not None else {} + ) + } + } + coordinator.session = parent_session + + config: dict = {"features": {}, "settings": settings or {"exclude_tools": []}} + tool = DelegateTool(coordinator, config) + return tool, spawn_fn + + +# --------------------------------------------------------------------------- +# T2.1 -- Default injection +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +class TestDefaultInjection: + async def test_settings_default_flows_to_orchestrator_config(self) -> None: + tool, spawn_fn = _make_tool( + settings={"exclude_tools": [], "max_llm_calls": 300} + ) + + await tool._spawn_new_session( + agent_name="test-agent", + instruction="do something", + context_depth="none", + context_scope="conversation", + context_turns=5, + provider_preferences=None, + hooks=None, + ) + + call_kwargs = spawn_fn.call_args.kwargs + assert call_kwargs["orchestrator_config"]["max_iterations"] == 300 + assert call_kwargs["orchestrator_config"]["budget_warn_ratio"] == 0.8 + + async def test_ships_dark_by_default_no_settings(self) -> None: + """Regression: with no max_llm_calls setting at all (S0 default), + orchestrator_config is untouched -- exactly today's behavior.""" + tool, spawn_fn = _make_tool() # no settings.max_llm_calls + + await tool._spawn_new_session( + agent_name="test-agent", + instruction="do something", + context_depth="none", + context_scope="conversation", + context_turns=5, + provider_preferences=None, + hooks=None, + ) + + call_kwargs = spawn_fn.call_args.kwargs + # No orchestrator config was inherited and no budget was injected -- + # None, exactly like before this feature existed. + assert call_kwargs["orchestrator_config"] is None + + +# --------------------------------------------------------------------------- +# T2.2 / T2.4 -- Per-call override / precedence +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +class TestPerCallOverride: + async def test_per_call_override_wins_over_default(self) -> None: + """T2.2 + T2.4: input max_llm_calls=600 beats settings default 300.""" + tool, spawn_fn = _make_tool( + settings={"exclude_tools": [], "max_llm_calls": 300} + ) + + result = await tool.execute( + { + "agent": "self", + "instruction": "do something", + "max_llm_calls": 600, + } + ) + + assert result.success is True + call_kwargs = spawn_fn.call_args.kwargs + assert call_kwargs["orchestrator_config"]["max_iterations"] == 600 + + +# --------------------------------------------------------------------------- +# T2.5 -- Opt-out +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +class TestOptOut: + async def test_zero_disables_budget_for_this_call(self) -> None: + tool, spawn_fn = _make_tool( + settings={"exclude_tools": [], "max_llm_calls": 300} + ) + + result = await tool.execute( + { + "agent": "self", + "instruction": "do something", + "max_llm_calls": 0, + } + ) + + assert result.success is True + call_kwargs = spawn_fn.call_args.kwargs + # No orchestrator config inherited from parent, and 0 means "no + # budget" -- orchestrator_config collapses back to None entirely. + assert call_kwargs["orchestrator_config"] is None + + +# --------------------------------------------------------------------------- +# T2.6 -- Parent config preserved, never mutated +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +class TestParentConfigPreserved: + async def test_other_keys_survive_and_parent_dict_not_mutated(self) -> None: + tool, spawn_fn = _make_tool( + settings={"exclude_tools": [], "max_llm_calls": 300}, + orchestrator_value={ + "type": "loop-basic", + "config": {"stream_delay": 0.05, "extended_thinking": True}, + }, + ) + parent_orch_config = tool.coordinator.session.config["session"]["orchestrator"][ + "config" + ] + original_parent_config = dict(parent_orch_config) + + await tool._spawn_new_session( + agent_name="test-agent", + instruction="do something", + context_depth="none", + context_scope="conversation", + context_turns=5, + provider_preferences=None, + hooks=None, + ) + + call_kwargs = spawn_fn.call_args.kwargs + sent = call_kwargs["orchestrator_config"] + assert sent["stream_delay"] == 0.05 + assert sent["extended_thinking"] is True + assert sent["max_iterations"] == 300 + # The parent's own config dict must be untouched (no new keys, no + # mutation) -- _spawn_new_session copies before adding budget keys. + assert parent_orch_config == original_parent_config + assert "max_iterations" not in parent_orch_config + + +# --------------------------------------------------------------------------- +# T2.7 -- Validation at construction, not at spawn +# --------------------------------------------------------------------------- + + +class TestValidation: + def test_negative_value_raises_at_construction(self) -> None: + coordinator = MagicMock() + with pytest.raises(ValueError, match="max_llm_calls"): + DelegateTool( + coordinator, + {"features": {}, "settings": {"max_llm_calls": -1}}, + ) + + def test_bool_raises_at_construction(self) -> None: + coordinator = MagicMock() + with pytest.raises(TypeError, match="bool"): + DelegateTool( + coordinator, + {"features": {}, "settings": {"max_llm_calls": True}}, + ) + + def test_string_raises_at_construction(self) -> None: + coordinator = MagicMock() + with pytest.raises(TypeError, match="max_llm_calls"): + DelegateTool( + coordinator, + {"features": {}, "settings": {"max_llm_calls": "300"}}, + ) + + def test_zero_and_none_both_collapse_to_none(self) -> None: + assert _validate_call_budget(0) is None + assert _validate_call_budget(None) is None + assert _validate_call_budget(300) == 300 + + +# --------------------------------------------------------------------------- +# T2.8 / T2.9 -- Status and metadata passthrough +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +class TestStatusAndMetadataPassthrough: + async def test_budget_exhausted_status_and_metadata_forwarded(self) -> None: + spawn_result = { + "output": "Summary: made progress, here is what remains.", + "session_id": "child-001", + "status": "budget_exhausted", + "turn_count": 300, + "metadata": { + "llm_calls": 300, + "llm_call_budget": 300, + "budget_exhausted": True, + "resumable": True, + }, + } + tool, _spawn_fn = _make_tool( + settings={"exclude_tools": [], "max_llm_calls": 300}, + spawn_result=spawn_result, + ) + + result = await tool.execute( + {"agent": "self", "instruction": "do a very large task"} + ) + + assert result.success is True + assert result.output is not None + assert result.output["status"] == "budget_exhausted" + assert result.output["metadata"]["llm_calls"] == 300 + assert result.output["metadata"]["llm_call_budget"] == 300 + assert result.output["metadata"]["budget_exhausted"] is True + assert result.output["metadata"]["resumable"] is True + # Budget was requested and the child reported llm_call_budget -- + # negotiated feature confirmed active. + assert result.output["metadata"]["budget_enforced"] is True + + +# --------------------------------------------------------------------------- +# T2.10 -- Negotiated-feature warning +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +class TestNegotiatedFeatureWarning: + async def test_missing_budget_telemetry_sets_budget_enforced_false( + self, caplog: pytest.LogCaptureFixture + ) -> None: + # Child's orchestrator doesn't implement max_iterations at all -- + # its metadata carries no llm_call_budget key. + spawn_result = { + "output": "done", + "session_id": "child-001", + "status": "success", + "turn_count": 1, + "metadata": {}, + } + tool, _spawn_fn = _make_tool( + settings={"exclude_tools": [], "max_llm_calls": 300}, + spawn_result=spawn_result, + ) + + with caplog.at_level("WARNING"): + result = await tool.execute( + {"agent": "self", "instruction": "do something"} + ) + + assert result.success is True + assert result.output is not None + assert result.output["metadata"]["budget_enforced"] is False + assert any( + "Layer 1 bounding is NOT active" in record.message + for record in caplog.records + ) + + async def test_budget_enforced_absent_when_no_budget_requested(self) -> None: + """When no budget was ever requested (ships dark, S0), the + budget_enforced key must not appear at all -- it's not a relevant + concept for a delegation with no Layer 1 budget.""" + tool, _spawn_fn = _make_tool() # no settings.max_llm_calls + + result = await tool.execute({"agent": "self", "instruction": "do something"}) + + assert result.success is True + assert result.output is not None + assert "budget_enforced" not in result.output["metadata"] + + +# --------------------------------------------------------------------------- +# T2.11 -- Resume carries no orchestrator_config +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +class TestResumeNoOrchestratorConfig: + async def test_resume_passes_no_orchestrator_config_kwarg(self) -> None: + resume_fn = AsyncMock( + return_value={ + "output": "continued", + "session_id": "child-001", + "status": "success", + "turn_count": 2, + "metadata": {}, + } + ) + coordinator = MagicMock() + coordinator.session_id = "parent-session-123" + coordinator.get_capability = lambda name: ( + resume_fn if name == "session.resume" else None + ) + coordinator.get = MagicMock(return_value=None) + + tool = DelegateTool( + coordinator, + { + "features": {}, + "settings": {"exclude_tools": [], "max_llm_calls": 300}, + }, + ) + + result = await tool.execute( + { + "session_id": "abc123-def456_self", + "instruction": "continue please", + } + ) + + assert result.success is True + # Resume's own signature has no orchestrator_config parameter at + # all -- the stored config (from the original spawn) already + # carries whatever budget was set. Confirm no such kwarg leaked in. + assert "orchestrator_config" not in resume_fn.call_args.kwargs + + +# --------------------------------------------------------------------------- +# Frontmatter round-trip verification (spec §16 open item #1) +# --------------------------------------------------------------------------- + + +class TestAgentFrontmatterBudgetGap: + def test_agent_frontmatter_budget_key_is_dropped(self) -> None: + """Verifies (does not merely assert from reading the source) that a + top-level `budget:` block in an agent .md file's frontmatter does + NOT survive into the dict `_load_agent_file_metadata` returns. + + This is the empirical gate the spec's open item #1 requires before + wiring precedence rank 2 (per-agent frontmatter override). It does + NOT round-trip today -- `_load_agent_file_metadata` only forwards a + fixed allowlist of top-level frontmatter keys (tools, providers, + hooks, session, provider_preferences, model_role, agents); `budget` + is not among them. Per the owner's decision, rank 2 is therefore + NOT implemented in this PR -- see the module README's "Known gaps" + section. `model_role` is asserted as a present-and-working control + to prove this isn't a wholesale frontmatter-loading failure. + + If this test starts failing (i.e. `budget` starts surviving), that + is the signal precedence rank 2 can finally be wired -- update this + test and the delegate's `_resolve_call_budget` together. + """ + from amplifier_foundation.bundle._dataclass import _load_agent_file_metadata + + content = """--- +meta: + name: test-agent + description: "A test agent" +model_role: research +budget: + max_llm_calls: 500 +--- + +Test instruction body. +""" + with tempfile.TemporaryDirectory() as d: + path = Path(d) / "test-agent.md" + path.write_text(content, encoding="utf-8") + result = _load_agent_file_metadata(path, "test-agent") + + assert "budget" not in result, ( + "budget now survives agent-frontmatter loading -- precedence " + "rank 2 (per-agent override) can be wired; update " + "_resolve_call_budget and this test together." + ) + # Control: model_role (an allowlisted key) DOES survive, proving + # this isn't a general frontmatter-loading failure. + assert result.get("model_role") == "research"