diff --git a/README.md b/README.md index 632c947..4fdd3f2 100644 --- a/README.md +++ b/README.md @@ -364,7 +364,9 @@ for the full write-up. Neither defect is inherited here: module = "context-simple" config = { compaction_strategy = "summary", # default: "progressive" - summary_call_mode = "standalone", # default; "fork" appends onto the live prefix + summary_call_mode = "standalone", # default; "fork" appends onto the live prefix, + # "auto" = fork gated by the span-size predicate + summary_fork_min_span_ratio = 0.22,# optional; None (default) = predicate off summary_trigger = 0.60, # usage fraction that starts the async summarizer summarization_model = "...", # optional; None uses the provider default summarization_prompt_path = "...", # optional file override for the 5-section prompt @@ -550,16 +552,122 @@ meter's comparand) does not move, and a forked session serves views bit-for-bit identical to an unforked control; tool-pair snapping is unchanged; every refusal falls back loudly. -**Not proven — no cache or cost win is claimed.** G-FORK-PREFIX / -G-FORK-CACHED / G-FORK-COST / G-FORK-NOBOUNDARY are a separately funded -evaluation; none of it was run here. +**And by `tests/test_summary_fork_span_predicate.py`** for the span-size +predicate below: the default is off in *both* fork modes; the threshold is +pinned from both sides against the fixture's own realized ratio (not a +guessed constant); a decline sends the standalone request byte-for-byte and +never moves the fallback counter; a misalignment is still a fallback even +with the predicate armed at a ratio nothing could fail. -**Honest ceiling, stated in advance:** this reduces the separate summarizer -charge only. It does **not** reduce the boundary rebuild, which §2d -measured as the dominant cost (+84% boundaries drives the +83% run cost). -If the summarizer really is 2.4% of run cost rather than 8.3–10.9%, then -even a perfect fork is worth ~2% — the arm design must therefore report the -summarizer's cost share from the run's own per-call table, not from prose. +**Now measured — and cost-neutral as shipped.** Lane `model_performance-6da` +ran all four gates end-to-end (S5-CRAC, n=3/arm balanced across two +containers, `gpt-5.6-terra@medium`); three of four FAILED. The cache +mechanism is real — a forked call reads a **median 85.7%** of its own prompt +from cache against **0.0%** for a standalone one — but total run cost moved +**−0.8%**, i.e. noise. Unconditional forking is not worth shipping. Why, and +the fix, is the next section. + +**One correction that outlived the treatment:** the summarizer is **~30% of +run cost** on this workload, not the 2.4%/8.3–10.9% previously recorded. +Both older figures were the same artefact — pairing each `llm:response` with +the most recent `llm:request` misfiles a summarizer call that ran +concurrently as a background task. Attribution must come from the module's +own `ChatResponse.usage`, never from positional pairing. + +**Honest ceiling, unchanged:** this reduces the separate summarizer charge +only. It does **not** reduce the boundary rebuild, which §2d measured as the +dominant cost (+84% boundaries drives the +83% run cost). + +## Span-size predicate in front of the fork (`summary_fork_min_span_ratio`) + +**Default off. `None` (or `0`) means the predicate is not evaluated, no +counter moves, and `summary_call_mode: "fork"` behaves exactly as it did +before this feature existed.** + +### Why unconditional forking nets to zero + +6da measured both halves of the trade: + +| measure | forked call | standalone call | +|---|---|---| +| $ per 1,000 own-prompt tokens (median) | **0.00077** | **0.00350** — fork is 4.5× cheaper per token | +| own prompt size (median tokens) | **26,620** | **5,129** — fork's prompt is 5.2× bigger | + +4.5× cheaper per token × 5.2× more tokens ≈ 1.0. The fork trades "a small, +wholly-uncached prompt" for "a large, mostly-cached prompt", and at these +sizes those cost the same. + +### But the trade is span-size dependent + +| population | n | mean cost | +|---|---|---| +| standalone, own prompt **> 30,000** tok | 17 | **$0.1410** | +| standalone, own prompt **≤ 15,000** tok | 40 | **$0.0151** | +| forked, any span | 20 | **$0.0265** — roughly flat | + +A forked call costs ~$0.027 no matter what it summarizes, because it always +pays for the conversation prefix. A standalone call costs almost nothing on +a small span and **5× a fork's price on a large one**. Fork wins on the tail +and loses on the median. + +### The threshold is derived, not chosen + +A standalone call pays for the **span** at the uncached rate; a fork pays for +the whole **prefix** at the cached rate. With `S` = span tokens and `P` = +prefix tokens, the fork is cheaper exactly when + +``` +P × 0.00077 < S × 0.00350 ⟺ S / P > 0.22 +``` + +`0.22` is that ratio and nothing else — it is `DEFAULT_FORK_MIN_SPAN_RATIO`, +and `summary_call_mode: "auto"` is the one-word way to say "fork, predicate +on, at that default". Two independent cross-checks against the same measured +table, both of which it passes: + +- 6da's median span (5,129 tok) over its median fork prompt (26,620 tok) is + **0.193** — just *below* break-even, which is precisely why 6da measured + the two arms cancelling "exactly" at the median. +- The bucket table: spans ≤15k imply a mean span ~4.3k → ratio ~0.16 → + **declines**, and standalone at $0.0151 does beat a fork at $0.0265. Spans + >30k imply a mean span ~40k → ratio ≥0.67 → **forks**, at a flat ~$0.027 + against $0.1410. Both verdicts match the money. + +**Why a ratio and not a token count.** Fork cost scales with the *prefix*, +which grows all session; standalone cost scales with the *span*. The +break-even is therefore a ratio of two rates, and a fixed token threshold is +only correct at one prefix size. (At 6da's median prefix the equivalent +absolute threshold is ~5,900 span tokens; late in a session the same ratio +is a much larger number.) + +### A decline is not a fallback + +The predicate is evaluated **strictly after** every alignment precondition, +so it only ever sees forks that would have worked, and declines them on cost +alone. That ordering is load-bearing in both directions: no span size can +buy a misaligned fork, and a misalignment is never misreported as an +economic decision. Declines move `fork_declines`; genuine refusals move +`fork_fallbacks`; `last_summary_call_stats` reports both separately, plus a +`span_measure` (`span_tokens`, `prefix_tokens`, `span_ratio`, +`min_span_ratio`) so an eval arm can plot the realized distribution and +re-derive its own threshold. One combined counter would make a healthy +predicate and a broken fork look identical. + +Declines log at `INFO`, once per kind — not `WARNING`, which in this module +means something is wrong. + +### The cap this does **not** lift + +6da also found the fork rate structurally capped at **45.5%** (20/44) on a +CLI workload: every turn is a fresh `amplifier run --resume` process, so +`note_request_sent()` has not been called when the first summary trigger of +the turn fires. **The predicate runs after that refusal and therefore cannot +raise the fork rate — only lower it.** Worse, the capped calls skew *large* +(mean $0.0872 against $0.0265 forked), i.e. exactly the population the +predicate wants to route to a fork. On a long-lived in-process session the +cap does not apply and the predicate captures the full tail; on a CLI +workload its realized win is bounded by the ≤45.5% forkable subset. Lifting +that cap is a separate change to the seam, not to this predicate. ## Tool-result budget, shape, and spill diff --git a/amplifier_module_context_simple/__init__.py b/amplifier_module_context_simple/__init__.py index 9fed112..e3d6a29 100644 --- a/amplifier_module_context_simple/__init__.py +++ b/amplifier_module_context_simple/__init__.py @@ -209,11 +209,59 @@ an explicit-breakpoint cache (Anthropic) needs, since the provider places its breakpoint at the last STABLE message -- i.e. exactly where this module's view ends. - • UNMEASURED. The gates (G-FORK-PREFIX / G-FORK-CACHED / G-FORK-COST / - G-FORK-NOBOUNDARY) are a separately funded eval; nothing in this - module claims a measured cache or cost win yet. What IS proven here - is structural: default byte-identity, main-line non-mutation, and - pure-append shape. + • NOW MEASURED, AND THE VERDICT IS "COST-NEUTRAL AS-IS". Lane 6da ran + the four gates end-to-end (S5-CRAC, n=3/arm, gpt-5.6-terra@medium) + and three of four FAILED. The cache mechanism is real -- a forked + call reads a median 85.7% of its own prompt from cache against + 0.0% for a standalone one -- but total run cost moved -0.8%, i.e. + noise. WHY it cancels, and what to do about it, is the next section. + What was already proven here is structural and still holds: default + byte-identity, main-line non-mutation, and pure-append shape. + +Span-Size Predicate In Front Of The Fork (opt-in, default off -- see config +`summary_fork_min_span_ratio`, and mode "auto"): + • THE PROBLEM IT SOLVES: fork mode as shipped forks EVERY summarization + it is aligned for, and lane 6da measured why that nets to zero. A + forked call is 4.5x cheaper per token ($0.00077 vs $0.00350 per 1k + own-prompt tokens) but its prompt is 5.2x bigger (median 26,620 vs + 5,129 tokens), because it always carries the whole conversation + prefix. 4.5x cheaper x 5.2x more tokens ~= 1.0. The mechanism works + and pays for itself exactly. + • THE TRADE IS SPAN-SIZE DEPENDENT, and that is the lever. Measured: + a standalone call on a span >30,000 tokens costs $0.1410 mean; on a + span <=15,000 tokens, $0.0151. A forked call is roughly FLAT at + $0.0265 regardless, because what it pays for is the prefix, not the + span. So the fork wins ~5x on the tail and loses on the median. + • THE FIX: gate the fork on span size RELATIVE TO PREFIX size. A + standalone call pays for the SPAN at the uncached rate; a fork pays + for the PREFIX at the cached rate. Fork is cheaper exactly when + span/prefix > (cached rate / uncached rate) = 0.22. That is + DEFAULT_FORK_MIN_SPAN_RATIO, derived from the measured rates rather + than chosen -- see its definition for the derivation and two + independent cross-checks against the same table. + • WHY A RATIO AND NOT A TOKEN COUNT. Fork cost scales with the PREFIX, + which grows all session; standalone cost scales with the SPAN. The + break-even is therefore a ratio of two rates, and a fixed token + threshold is only correct at one prefix size. (At 6da's median + prefix the equivalent absolute threshold is ~5,900 span tokens; by + late session the same ratio is a much larger number.) + • `summary_fork_min_span_ratio: None` (DEFAULT) or `0` disables the + predicate entirely: not evaluated, no counter moved, fork mode + behaves exactly as it did before this feature. `summary_call_mode: + "auto"` is the one-word way to say "fork, predicate on, at the + measured default". An explicit ratio always wins over both. + • A DECLINE IS NOT A FALLBACK. The predicate is evaluated strictly + AFTER every alignment precondition, so it only ever sees forks that + would have worked, and it declines them on cost alone. Declines move + `fork_declines`; genuine refusals move `fork_fallbacks`; the two are + reported separately in `last_summary_call_stats` because a healthy + predicate and a broken fork would otherwise look identical. + • THE CAP THIS DOES NOT LIFT. 6da also found the fork rate structurally + capped at 45.5% (20/44) on a CLI workload: every turn is a fresh + process, so `note_request_sent()` has not been called when the first + summary trigger of the turn fires. The predicate runs after that + refusal and therefore CANNOT raise the fork rate -- only lower it. + It captures the tail win within whatever subset is forkable at all. Tool-result budget and spill (opt-in, default off -- see config `tool_result_budget_tokens` / `tool_result_shape` / @@ -360,10 +408,62 @@ # "inline" is accepted as an alias for "standalone": the lane brief that # commissioned this work named the default mode "inline" while the work # item named it "standalone". Both mean "today's behavior, unchanged". +# +# "auto" is "fork, but only when the span-size predicate says forking is +# actually cheaper" -- see DEFAULT_FORK_MIN_SPAN_RATIO and the module +# docstring "Span-size predicate in front of the fork". SUMMARY_CALL_MODE_STANDALONE = "standalone" SUMMARY_CALL_MODE_FORK = "fork" -_VALID_SUMMARY_CALL_MODES = (SUMMARY_CALL_MODE_STANDALONE, SUMMARY_CALL_MODE_FORK) +SUMMARY_CALL_MODE_AUTO = "auto" +_VALID_SUMMARY_CALL_MODES = ( + SUMMARY_CALL_MODE_STANDALONE, + SUMMARY_CALL_MODE_FORK, + SUMMARY_CALL_MODE_AUTO, +) _SUMMARY_CALL_MODE_ALIASES = {"inline": SUMMARY_CALL_MODE_STANDALONE} +# The modes under which a fork is even attempted. "auto" differs from +# "fork" ONLY in which default the span-size predicate resolves to; every +# alignment precondition is identical and is checked first. +_FORK_CALL_MODES = (SUMMARY_CALL_MODE_FORK, SUMMARY_CALL_MODE_AUTO) + +# Break-even span:prefix ratio for the fork, DERIVED (not chosen) from the +# per-token rates lane 6da measured end-to-end on S5-CRAC: +# +# forked call: $0.00077 per 1,000 own-prompt tokens +# standalone call: $0.00350 per 1,000 own-prompt tokens +# +# A standalone call's own prompt IS the span (its own ~955-char system +# prompt plus a fresh rendering of the span). A forked call's own prompt is +# the whole PREFIX -- which already contains the span; the appended +# instruction is a rounding error. So, with S = span tokens and P = prefix +# tokens, the fork is cheaper exactly when +# +# P x 0.00077 < S x 0.00350 <=> S / P > 0.22 +# +# 0.22 is that ratio and nothing else. Two independent cross-checks against +# the same measured table, both of which it passes: +# +# • 6da's median span (5,129 tok) over its median fork prompt (26,620 +# tok) is 0.193 -- just BELOW break-even, which is precisely why 6da +# measured the two arms cancelling "exactly" at the median. +# • 6da's span buckets: standalone spans <=15k cost $0.0151 mean (implied +# mean span ~4.3k, ratio ~0.16 -> predicate DECLINES, standalone wins, +# correct); standalone spans >30k cost $0.1410 mean (implied mean span +# ~40k, ratio >=0.67 -> predicate FORKS at a flat ~$0.027, correct, a +# 5.3x win). +# +# Evidence: .amplifier/evaluation/probes/6da-summary-fork/FINDINGS.md §6 +# ("WHY G-FORK-COST FAILED -- the two effects cancel, exactly") in the +# openai-evals-team-ci repo. It is a BREAK-EVEN, not a margin: it is the +# point where the two calls cost the same. Raising it buys margin, lowering +# it forks speculatively; both are one config value away. +DEFAULT_FORK_MIN_SPAN_RATIO = 0.22 + +# There is exactly one kind of predicate decline, and its message carries +# per-call token counts. Deduping the announcement on the MESSAGE would +# therefore dedupe nothing; it dedupes on this constant instead. See +# _note_fork_declined. +_FORK_DECLINE_KIND = "span-below-min-span-ratio" # How much of the span's final message is quoted back in the fork # instruction as the "summarize up to HERE" boundary marker. Long enough to @@ -505,7 +605,7 @@ def _is_real_user_message(entry: dict[str, Any]) -> bool: def _normalize_summary_call_mode(value: Any) -> str: """Canonicalize a `summary_call_mode` config value. - Accepts the two real modes plus the documented "inline" alias for + Accepts the three real modes plus the documented "inline" alias for "standalone" (the commissioning lane brief and the work item named the same default differently -- see _SUMMARY_CALL_MODE_ALIASES). An unusable value logs a warning and falls back to "standalone", matching @@ -524,6 +624,45 @@ def _normalize_summary_call_mode(value: Any) -> str: return mode +def _normalize_fork_min_span_ratio(value: Any) -> float | None: + """Canonicalize a `summary_fork_min_span_ratio` config value. + + None (and 0, mirroring how `compact_clear_at_least` spells "off") + disables the predicate entirely: it is not evaluated, no counter moves, + and fork mode behaves exactly as it did before this feature existed. + Otherwise a positive real number, interpreted as span_tokens / + prefix_tokens. + + Values > 1.0 are legal and meaningful -- the span is a SUBSET of the + prefix, so a ratio above 1.0 can never be met and is a deliberate "arm + the plumbing, never fork" setting. An unusable value (non-numeric, + negative, NaN) logs a warning and disables the predicate, matching how + every other knob in this module is validated: refusing to start is + worse than running at the old default and saying so. + """ + if value is None: + return None + try: + ratio = float(value) + except (TypeError, ValueError): + logger.warning( + f"context-simple: unusable summary_fork_min_span_ratio {value!r} " + "(expected None or a positive number); the span-size predicate " + "is DISABLED and fork mode will fork whenever it is aligned" + ) + return None + if ratio != ratio or ratio < 0: # NaN or negative + logger.warning( + f"context-simple: unusable summary_fork_min_span_ratio {value!r} " + "(expected None or a positive number); the span-size predicate " + "is DISABLED and fork mode will fork whenever it is aligned" + ) + return None + if ratio == 0: + return None + return ratio + + async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = None): """ Mount the simple context manager. @@ -582,15 +721,27 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = is the cheapest lever; see README "Known issue: boundary refire". - summary_call_mode: "standalone" (default, byte-identical to the - pre-existing summarizer call) or "fork" -- issue the + pre-existing summarizer call), "fork" -- issue the summarization ask as a pure append onto the prefix the main line already sent, so it can be cache-read instead of paying - fresh input tokens for the span. "inline" is accepted as an - alias for "standalone". Only consulted when - compaction_strategy == "summary". Fork mode additionally - requires the caller to have called note_request_sent(); see - module docstring "Cache-safe forking of the summarization - call". + fresh input tokens for the span -- or "auto", which is "fork" + gated by the span-size predicate at its measured break-even + default. "inline" is accepted as an alias for "standalone". + Only consulted when compaction_strategy == "summary". Fork + mode additionally requires the caller to have called + note_request_sent(); see module docstring "Cache-safe forking + of the summarization call". + - summary_fork_min_span_ratio: Span-size predicate in front of + the fork. None (default) or 0 disables it entirely -- fork + mode then forks whenever it is aligned, exactly as before. + A positive number is the minimum span_tokens/prefix_tokens + at which forking is worth it; below it the summarizer runs + standalone BY DESIGN (recorded, not counted as a fallback). + Unset under summary_call_mode "auto" it resolves to + DEFAULT_FORK_MIN_SPAN_RATIO (0.22, the break-even derived + from lane 6da's measured per-token rates). Only consulted + when a fork is otherwise possible; see module docstring + "Span-size predicate in front of the fork". - summarization_model: Model identifier passed to the summarizer's ChatRequest (default: None, i.e. provider default). Setting it CONFLICTS with summary_call_mode "fork" (a summarizer routed to @@ -684,6 +835,7 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = compaction_strategy=compaction_strategy, summary_trigger=config.get("summary_trigger", 0.60), summary_call_mode=summary_call_mode, + summary_fork_min_span_ratio=config.get("summary_fork_min_span_ratio"), summarization_model=config.get("summarization_model"), summarization_prompt_path=config.get("summarization_prompt_path"), summarization_timeout_s=config.get("summarization_timeout_s", 30.0), @@ -776,6 +928,7 @@ def __init__( compaction_strategy: str = COMPACTION_STRATEGY_PROGRESSIVE, summary_trigger: float = 0.60, summary_call_mode: str = SUMMARY_CALL_MODE_STANDALONE, + summary_fork_min_span_ratio: float | None = None, summarization_model: str | None = None, summarization_prompt_path: str | None = None, summarization_timeout_s: float = 30.0, @@ -842,15 +995,24 @@ def __init__( module docstring and README "Known issue: boundary refire". summary_call_mode: "standalone" (default; the pre-existing - two-message summarizer call, byte-identical) or "fork" (the + two-message summarizer call, byte-identical), "fork" (the same ask appended onto the main line's already-sent prefix - so the provider can cache-read it). "inline" is an accepted - alias for "standalone". Only consulted when - compaction_strategy == "summary". See module docstring - "Cache-safe forking of the summarization call" -- in - particular, fork mode needs note_request_sent() to have been - called, and falls back to standalone (loudly) when it has - not. + so the provider can cache-read it), or "auto" ("fork" with + the span-size predicate on at its measured break-even + default). "inline" is an accepted alias for "standalone". + Only consulted when compaction_strategy == "summary". See + module docstring "Cache-safe forking of the summarization + call" -- in particular, fork mode needs note_request_sent() + to have been called, and falls back to standalone (loudly) + when it has not. + summary_fork_min_span_ratio: Minimum span_tokens/prefix_tokens + at which a fork is worth issuing. None (default) or 0 + disables the predicate entirely -- not evaluated, no + counter moved, fork mode unchanged from before this + feature. Unset under summary_call_mode "auto" it resolves + to DEFAULT_FORK_MIN_SPAN_RATIO. An explicit value always + wins, in either fork mode. See module docstring "Span-size + predicate in front of the fork". summarization_model: Model identifier for the summarizer's own ChatRequest. None uses the provider's default model. Incompatible with summary_call_mode "fork" -- a summarizer @@ -916,6 +1078,13 @@ def __init__( compaction_strategy = COMPACTION_STRATEGY_PROGRESSIVE self.compaction_strategy = compaction_strategy self.summary_call_mode = _normalize_summary_call_mode(summary_call_mode) + # None means "predicate off". Kept as the RAW normalized config so + # `_effective_fork_min_span_ratio` can still tell "the caller said + # nothing" from "the caller said 0.22", which is the whole + # difference between mode "fork" and mode "auto". + self.summary_fork_min_span_ratio = _normalize_fork_min_span_ratio( + summary_fork_min_span_ratio + ) self.summary_trigger = summary_trigger self.summarization_model = summarization_model self.summarization_prompt_path = summarization_prompt_path @@ -1024,9 +1193,27 @@ def __init__( # these to tell a real fork from a silently unforked one. self._last_summary_call: dict[str, Any] | None = None self._summary_fork_fallbacks: int = 0 + # DELIBERATE declines by the span-size predicate, counted SEPARATELY + # from `_summary_fork_fallbacks` on purpose. A fallback means "a + # fork was wanted and could not be done" -- a defect signal an eval + # arm reads to detect silent unforking. A decline means "a fork was + # possible and the predicate judged it more expensive than the + # standalone call" -- the feature working. Summing them into one + # number would make a correctly-working predicate look exactly like + # a broken fork. + self._summary_fork_declines: int = 0 + # What the predicate actually measured on the last summarizer call + # (None when it was not consulted). Surfaced through + # last_summary_call_stats so an eval arm can plot the realized + # span:prefix distribution and re-derive the threshold from its own + # data instead of trusting this module's default. + self._last_fork_span_measure: dict[str, Any] | None = None # Precondition names already warned about, so a session that can # never fork logs once per reason instead of once per summarization. self._fork_warned: set[str] = set() + # Same once-per-reason discipline for predicate declines, which are + # announced at INFO (they are by design, not a problem). + self._fork_declined_warned: set[str] = set() # Real-usage token meter state (see _on_llm_response / # _measure_working_tokens). `_last_measured_prompt_tokens` holds the # most recent real usage observed via `llm:response` @@ -1220,9 +1407,19 @@ def last_summary_call_stats(self) -> dict[str, Any] | None: None before the first one. Otherwise a dict with `mode_requested`, `mode_used`, `reason` (None when the requested mode was honored), - `prefix_messages`, and `fork_fallbacks` (session-cumulative). This - is how an eval arm distinguishes a real fork from a silently - unforked one WITHOUT patching the module. + `prefix_messages`, `fork_fallbacks` (session-cumulative), + `fork_declines` (session-cumulative, and NOT the same thing -- see + below), and `span_measure` (None unless the span-size predicate was + consulted; otherwise `span_tokens`, `prefix_tokens`, `span_ratio`, + `min_span_ratio`). + + This is how an eval arm distinguishes a real fork from a silently + unforked one WITHOUT patching the module -- and, since the + predicate landed, a DECLINED fork (the predicate judging standalone + cheaper: working as designed) from a REFUSED one (a fork that was + wanted and could not be built: a defect signal). `span_measure` + additionally lets an arm plot the realized span:prefix distribution + and re-derive its own threshold from its own data. """ return dict(self._last_summary_call) if self._last_summary_call else None @@ -4170,7 +4367,105 @@ def _fork_armed(self) -> bool: """True only when a forked summarizer call is actually configured.""" return ( self.compaction_strategy == COMPACTION_STRATEGY_SUMMARY - and self.summary_call_mode == SUMMARY_CALL_MODE_FORK + and self.summary_call_mode in _FORK_CALL_MODES + ) + + def _effective_fork_min_span_ratio(self) -> float | None: + """The span:prefix ratio this configuration actually gates on. + + An explicit `summary_fork_min_span_ratio` always wins, in either + fork mode -- including under "auto", where setting it explicitly is + how you override the measured default. Left unset, "auto" resolves + to DEFAULT_FORK_MIN_SPAN_RATIO and plain "fork" resolves to None + (predicate off, PR #27 behavior unchanged). + """ + if self.summary_fork_min_span_ratio is not None: + return self.summary_fork_min_span_ratio + if self.summary_call_mode == SUMMARY_CALL_MODE_AUTO: + return DEFAULT_FORK_MIN_SPAN_RATIO + return None + + def _fork_span_measure( + self, span: list[dict[str, Any]], prefix: list[dict[str, Any]] + ) -> tuple[int, int, float | None]: + """(span_tokens, prefix_tokens, ratio) for the predicate. + + Both sides use this module's own `_estimate_tokens`, the same unit + the rest of the compaction ladder is denominated in. The predicate + compares two counts produced by ONE estimator, so its verdict is + exact in its own units and never mixes an estimate with a + provider-reported figure. Ratio is None when the prefix measures + zero (nothing to divide by; the caller treats that as "no opinion"). + """ + span_tokens = self._estimate_tokens(span) + prefix_tokens = self._estimate_tokens(prefix) + if prefix_tokens <= 0: + return span_tokens, prefix_tokens, None + return span_tokens, prefix_tokens, span_tokens / prefix_tokens + + def _fork_span_declined_reason( + self, + span_tokens: int, + prefix_tokens: int, + ratio: float | None, + threshold: float, + ) -> str | None: + """Why this fork is DECLINED as not worth issuing, or None to fork. + + This is an economic judgement, not an alignment check -- it runs + strictly AFTER every `_fork_refusal_reason` branch, so by the time + it is consulted the fork is known to be correct and would work. + The question here is only whether it is cheaper. + + THE ECONOMICS, in one line: a standalone call pays for the SPAN at + the uncached rate; a fork pays for the whole PREFIX at the cached + rate. Fork wins when span/prefix exceeds the ratio of those two + rates -- see DEFAULT_FORK_MIN_SPAN_RATIO for the measured + derivation. Below that line the fork is not merely a smaller win, + it is a LOSS, which is why it declines rather than shrugging. + """ + if ratio is None: + # Prefix measured zero tokens. Nothing sane to divide by, and a + # zero-token prefix is not a prefix worth protecting from a + # cache rebuild either. Say nothing; let the fork proceed. + return None + if ratio >= threshold: + return None + return ( + f"the span is {span_tokens} tok against a {prefix_tokens} tok " + f"prefix (ratio {ratio:.3f}), below the " + f"summary_fork_min_span_ratio of {threshold:.3f} -- forking " + "would re-pay for the whole prefix to save re-sending a span " + "smaller than that costs" + ) + + def _note_fork_declined(self, reason: str) -> None: + """Record and (once per decline KIND) announce a DECLINED fork. + + Deliberately NOT `_note_fork_fallback`: this is the predicate doing + its job, so it moves its own counter and speaks at INFO rather than + WARNING. Conflating the two would make a working predicate + indistinguishable from the silent-unforking defect fork mode's + warnings exist to surface. + + Note the dedupe key is the decline KIND, not the message. Every + decline message carries its own token counts, so deduping on the + message would dedupe nothing and emit a line per summarization -- + which is precisely the log-spam `_note_fork_fallback`'s + once-per-reason discipline exists to avoid. The numbers still reach + anyone who wants them, on the DEBUG line and in + `last_summary_call_stats["span_measure"]`. + """ + self._summary_fork_declines += 1 + if _FORK_DECLINE_KIND in self._fork_declined_warned: + logger.debug(f"context-simple: summarizer fork declined again ({reason})") + return + self._fork_declined_warned.add(_FORK_DECLINE_KIND) + logger.info( + f"context-simple: summary_call_mode={self.summary_call_mode!r} could " + f"have forked this summarization but DECLINED -- {reason}. The " + "standalone call is the cheaper one here; this is the span-size " + "predicate working, not a failure. Logged once per distinct reason." ) def _capture_fork_prefix(self) -> list[dict[str, Any]] | None: @@ -4372,25 +4667,55 @@ def _build_summary_request( reason: str | None = None if self._fork_armed(): + # Cleared inside the armed branch, never outside it: the default + # path must not so much as write a fork attribute (Group A). + self._last_fork_span_measure = None reason = self._fork_refusal_reason(messages_to_summarize, fork_prefix) if reason is None: - try: - assert fork_prefix is not None # guaranteed by the check above - return ( - self._build_fork_request( - messages_to_summarize, fork_prefix, prompt - ), - SUMMARY_CALL_MODE_FORK, - None, + assert fork_prefix is not None # guaranteed by the check above + # ORDER MATTERS. Alignment first (above), economics second + # (here). A misaligned fork is WRONG at any span size, so it + # must never be reachable by making the span big enough; and + # the predicate needs a real prefix to measure against, which + # the alignment checks are what guarantee. + threshold = self._effective_fork_min_span_ratio() + if threshold is not None: + span_tokens, prefix_tokens, ratio = self._fork_span_measure( + messages_to_summarize, fork_prefix ) - except Exception as e: - # A prefix message this module never created (unexpected - # content shape from a caller, say) can fail Message - # validation. Falling back keeps the summary happening at - # today's cost instead of turning a cache optimization - # into a lost summary. - reason = f"the forked request could not be built ({e!r})" - self._note_fork_fallback(reason) + self._last_fork_span_measure = { + "span_tokens": span_tokens, + "prefix_tokens": prefix_tokens, + "span_ratio": ratio, + "min_span_ratio": threshold, + } + declined = self._fork_span_declined_reason( + span_tokens, prefix_tokens, ratio, threshold + ) + if declined is not None: + self._note_fork_declined(declined) + # Return through the standalone branch below WITHOUT + # `_note_fork_fallback`: nothing went wrong here. + reason = declined + if reason is None: + try: + return ( + self._build_fork_request( + messages_to_summarize, fork_prefix, prompt + ), + SUMMARY_CALL_MODE_FORK, + None, + ) + except Exception as e: + # A prefix message this module never created + # (unexpected content shape from a caller, say) can + # fail Message validation. Falling back keeps the + # summary happening at today's cost instead of + # turning a cache optimization into a lost summary. + reason = f"the forked request could not be built ({e!r})" + self._note_fork_fallback(reason) + else: + self._note_fork_fallback(reason) formatted = self._format_messages_for_summarization(messages_to_summarize) request = ChatRequest( @@ -4475,6 +4800,16 @@ async def _run_summary_compaction_task( else 0 ), "fork_fallbacks": self._summary_fork_fallbacks, + # Separate from fork_fallbacks on purpose -- see + # `_summary_fork_declines`. A run with declines>0 and + # fallbacks==0 is a healthy predicate; the reverse is a + # wiring defect. One number could not say which. + "fork_declines": self._summary_fork_declines, + "span_measure": ( + dict(self._last_fork_span_measure) + if self._last_fork_span_measure + else None + ), } if self._hooks is not None: diff --git a/probes/pmt-fork-span-predicate/DONE-NOTE.md b/probes/pmt-fork-span-predicate/DONE-NOTE.md new file mode 100644 index 0000000..37b8fc5 --- /dev/null +++ b/probes/pmt-fork-span-predicate/DONE-NOTE.md @@ -0,0 +1,266 @@ +# DONE-NOTE — `model_performance-pmt` + +**Subject:** context-simple — gate `summary_call_mode` fork behind a span-size +predicate (fork only when the span is large relative to the prefix). + +**Branch:** `lane/pmt-fork-span-predicate` · **Base:** `main@a877b36` (the PR #27 +merge commit lane 6da measured) · **Spend: $0.00.** No API calls, no DTU, no +eval runs, no infrastructure created. 6da's measurement already exists and is +cited by file throughout. + +--- + +## 1. What shipped + +Three additions, all default no-op: + +| knob | default | meaning | +|---|---|---| +| `summary_fork_min_span_ratio` | `None` | minimum `span_tokens / prefix_tokens` at which a fork is worth issuing. `None` or `0` = predicate not evaluated at all | +| `summary_call_mode: "auto"` | — | `"fork"` with the predicate on at the measured break-even default | +| `DEFAULT_FORK_MIN_SPAN_RATIO` | `0.22` | the break-even, derived below | + +`summary_call_mode` is now `standalone` (a.k.a. `inline`) | `fork` | `auto`, +matching the item's requested vocabulary. **Two defaults, not one, are held +still:** `standalone` is byte-identical as before, *and* plain `fork` without +an explicit ratio still forks unconditionally, exactly as PR #27 shipped it. A +predicate that quietly switched itself on for existing `fork` users would be a +behaviour change wearing an opt-in's clothes. + +Observability: `last_summary_call_stats` gains `fork_declines` +(session-cumulative) and `span_measure` (`span_tokens`, `prefix_tokens`, +`span_ratio`, `min_span_ratio`), both alongside the pre-existing +`fork_fallbacks`. + +--- + +## 2. DELIVERABLES + +| deliverable | status | +|---|---| +| DRAFT PR on origin, branch `lane/pmt-fork-span-predicate`, default byte-identical, tests green | **DONE** — see §6 | +| threshold justified from 6da's measured span-cost table, cited by file | **DONE** — §3 | +| statement of whether the 45.5% fork-rate cap applies | **DONE** — §5. It applies, and strictly | +| DONE-NOTE.md under this lane's own dir, reproduced in the PR body, $0 spend | **DONE** — this file | + +--- + +## 3. The threshold: derived, not chosen + +**Source: `.amplifier/evaluation/probes/6da-summary-fork/FINDINGS.md` §6 +("WHY G-FORK-COST FAILED — the two effects cancel, exactly"), in the +`openai-evals-team-ci` repo.** + +6da's two measured per-token rates: + +``` +forked call: $0.00077 per 1,000 own-prompt tokens +standalone call: $0.00350 per 1,000 own-prompt tokens +``` + +The cost model those rates imply is asymmetric, and that asymmetry is the whole +finding: + +- a **standalone** call's own prompt *is the span* (its ~955-char system prompt + plus a fresh rendering of the span). Its cost scales with **S**, the span. +- a **forked** call's own prompt is *the whole prefix* — which already contains + the span; the appended instruction is a rounding error. Its cost scales with + **P**, the prefix. + +So the fork is cheaper exactly when + +``` +P × 0.00077 < S × 0.00350 ⟺ S / P > 0.22 +``` + +**0.22 is that ratio and nothing else.** It is a *break-even*, not a margin: +the point where the two calls cost the same. Raising it buys margin, lowering it +forks speculatively — both are one config value away, and neither is a number I +invented. + +### Two independent cross-checks against the same table + +**(a) The median cancellation.** 6da's median span is 5,129 tok and its median +fork prompt is 26,620 tok → ratio **0.193**, just *below* break-even. That is +exactly why 6da measured the two arms cancelling "exactly" at the median and +recorded a −0.8% (noise) total run cost. A break-even sitting a hair above the +observed median ratio is what that observation predicts. + +**(b) The bucket table.** 6da's span buckets, and what the predicate says about +each: + +| population | 6da mean cost | implied mean span | ratio vs median prefix | predicate says | correct? | +|---|---|---|---|---|---| +| standalone, span ≤15k (n=40) | $0.0151 | ~4,300 tok | ~0.16 | **decline** → standalone | ✅ $0.0151 < $0.0265 | +| standalone, span >30k (n=17) | $0.1410 | ~40,000 tok | ≥0.67 | **fork** → flat ~$0.027 | ✅ 5.3× win | +| forked, any span (n=20) | $0.0265 (flat) | — | — | — | — | + +Implied mean spans are `mean cost ÷ $0.00350 per 1k` — the standalone rate +from the same table. Both measured buckets are classified the way the money +went. This is pinned as a test +(`test_the_default_classifies_6das_own_measured_populations_correctly`), so a +future edit to the constant has to argue with the measurement. + +### Why a ratio and not an absolute token threshold + +The item offered either. A ratio is the correct primitive and an absolute +threshold is only correct at one prefix size: fork cost scales with **P**, +which grows all session, while standalone cost scales with **S**. The +break-even is a ratio of two *rates*, so the predicate must be a ratio. At +6da's median prefix the equivalent absolute threshold is ~5,900 span tokens; by +late session the same ratio is a much larger number. This also matches the +item's own acceptance criteria wording ("a configurable fraction of the +recorded prefix tokens"). + +--- + +## 4. A decline is not a fallback + +The predicate is evaluated **strictly after** every existing alignment +precondition, never before. That ordering is load-bearing in both directions: + +- no span size can buy a *misaligned* fork (a misaligned fork is wrong at any + size — it pays for the whole conversation as fresh input); +- a misalignment is never misreported as an economic decision. + +Consequently the two counters answer different questions and are kept apart: + +| counter | meaning | what a nonzero value tells you | +|---|---|---| +| `fork_fallbacks` | a fork was **wanted and could not be done** | wiring/alignment defect — the silent-unforking signal 6da relied on | +| `fork_declines` | a fork was **possible and judged more expensive** | the predicate working | + +Summing them into one number would make a healthy predicate look exactly like a +broken fork. Declines log at `INFO` (once per kind, because every decline +message carries its own token counts and message-level dedupe would dedupe +nothing); refusals keep their existing `WARNING`. + +--- + +## 5. Does the 45.5% fork-rate cap apply? **Yes — and strictly.** + +6da found 20 of 44 treatment-arm summarizer calls actually forked (45.5%), the +other 24 refused in two families — 12 × `note_request_sent() has never been +called`, 8 × `the span … is not present in the recorded prefix`. The first is +structural on a CLI workload: every turn is a fresh `amplifier run --resume` +process, and the summary trigger fires inside the first +`get_messages_for_request()`, strictly before that process has sent anything. + +**The predicate is subject to that same cap, and cannot relieve it.** Because +it runs *after* the refusal checks, it can only ever decline forks that were +already possible. It lowers the realized fork rate; it can never raise it. The +45.5% is an upper bound on what the predicate has any say over. + +**And the cap bites precisely where the predicate would have helped most.** +6da's own numbers: the treatment arm's fallbacks (n=17) cost **$0.08719 mean** +against **$0.02652** for forked calls — the refusals land on the large spans +that accumulate while a fork is impossible. Those are exactly the +high-ratio calls the predicate is designed to route *to* a fork, and they never +reach it. + +So, stated plainly for the PR: **on a CLI workload this predicate's realized win +is bounded by the ≤45.5% forkable subset, and the expensive tail is +disproportionately outside that subset. On a long-lived in-process session +(where `note_request_sent()` fires before the first trigger) the cap does not +apply and the predicate captures the full tail.** Lifting the cap is a change to +the seam — arming `note_request_sent()` earlier, or persisting the sent-tools +fact across a resume — not to this predicate, and it would multiply this +predicate's value rather than substitute for it. + +--- + +## 6. Tests + +New file: `tests/test_summary_fork_span_predicate.py` — **31 tests**, four +groups, written against how this can silently go wrong rather than how it is +supposed to work: + +- **A — the default must not move, in *both* fork modes.** `standalone` never + evaluates the predicate or writes a predicate attribute; `fork` without an + explicit ratio records no measurement and forks as before; `auto` resolves to + the default; an explicit ratio overrides `auto`; bad values disable loudly, + `0` disables silently; `mount()` threads both knobs. +- **B — the predicate fires above the threshold and not below.** The boundary is + pinned from **both sides against the fixture's own realized ratio** + (`_measure_realized_ratio()`, 0.498 in this fixture), separated by a single + epsilon — a predicate tested only at 0.000001 and 99.0 would pass while being + an order of magnitude wrong. Plus the 6da-population replay from §3(b), the + `>=` inclusive-boundary semantics, a zero-token prefix (no divide-by-zero, no + opinion), and ratios >1.0 as a legal "never fork" setting. +- **C — a decline is not a fallback.** Neither counter contaminates the other, + in either direction; a misalignment is still a fallback *with the predicate + armed at a ratio nothing could fail*, and records no `span_measure` (so the + distribution an eval arm plots is not contaminated with calls the threshold + never governed); declines log `INFO`, never `WARNING`, once per kind. +- **D — nothing else moves.** A declined fork sends the standalone request + **byte-for-byte** (digest-compared against a control manager that was never in + fork mode), still produces the summary, consumes no `_seq`, touches no + history, selects the same span with tool-pairs intact, and leaves the next + served view byte-identical. + +**Full suite: 312 passed, 1 skipped** (baseline before this change: 281 passed, +1 skipped). `ruff check`: clean. + +### The tests were mutation-checked, not just run + +A suite that passes first try proves nothing until you make it fail on purpose. +Four mutations, each reverted after: + +| mutation | caught by | +|---|---| +| predicate always allows (`if ratio >= threshold` → `if True`) | **10 tests** | +| `auto` no longer resolves to the default | 4 tests | +| a decline increments `fork_fallbacks` instead of `fork_declines` | 3 tests | +| `DEFAULT_FORK_MIN_SPAN_RATIO` changed 0.22 → 0.9 | 2 tests | + +The ordering guarantee (alignment before economics) is covered by +`test_a_misalignment_is_still_a_fallback_even_with_the_predicate_on`, which +asserts `span_measure is None` on a misaligned call — moving the predicate +earlier populates it and fails the test. + +### One honesty note on the suite + +`tests/test_compaction_performance.py::test_compaction_scales_sub_quadratically` +failed **once**, during a run executed concurrently with a mutation pass on the +same machine. It is a wall-clock ratio assertion (`large/small < 8`) and is +load-sensitive by construction. It passed 3/3 in isolation and 3/3 in +back-to-back full-suite runs on an unloaded machine. Pre-existing flake under +load, unrelated to this change — recorded rather than quietly re-run. + +--- + +## 7. What is NOT claimed + +- **No new measurement.** This lane spent $0 and ran no eval. The predicate's + *economic* justification is entirely 6da's measurement; what is proven *here* + is structural — default byte-identity in both modes, correct classification of + 6da's own two measured populations, and the counter/ordering separation. +- **No claimed win on the item's A/B acceptance criterion.** The item's second + criterion ("summarizer cost share falls measurably vs plain fork mode AND + total run cost does not regress, both terms signed") requires an n≥3/arm + S5-CRAC run that this lane's $0 authority does not fund. That criterion is + **NOT-POSSIBLE at this budget** and is left open, deliberately and on the + record, rather than substituted with an arithmetic counterfactual dressed up + as a result. +- **What the A/B would need to be valid**, so it is cheap to commission: two + arms differing *only* in `summary_fork_min_span_ratio` (off vs 0.22), + per-summarizer-call attribution taken from the module's own + `ChatResponse.usage` — **never positional `llm:request`/`llm:response` + pairing**, which is the defect that produced the wrong 2.4%/8.3–10.9% figures + — and `b_constraints` / `c_post_compaction` reported per arm so a cost win + bought with a retention loss cannot be reported as a win. The new + `span_measure` field exists precisely so that arm can plot the realized + span:prefix distribution and re-derive its own threshold from its own data + instead of trusting this module's default. + +## 8. Open / recommended next + +1. **Commission the A/B above** (the only thing standing between this and a + measured verdict). +2. **Lift the 45.5% cap** — persist the sent-tools fact across a `--resume`, or + arm `note_request_sent()` before the first `get_messages_for_request()`. + Independent of this predicate and worth more *because* of it: 6da measured + the capped calls at $0.0872 mean, the most expensive population in the run. +3. `00-what-we-know.md` §2h conflict #8 still records 2.4% vs 8.3–10.9%. 6da + filed `PROPOSED-CORRECTIONS.md` beside its FINDINGS; the real figure is + ~30%. Not this lane's file to edit (lane rule 2), noted so it is not lost. diff --git a/tests/test_summary_fork_span_predicate.py b/tests/test_summary_fork_span_predicate.py new file mode 100644 index 0000000..7dedf44 --- /dev/null +++ b/tests/test_summary_fork_span_predicate.py @@ -0,0 +1,654 @@ +"""Adversarial tests for the span-size predicate in front of the summarizer fork. + +WHAT MEASUREMENT THIS FILE EXISTS TO ENCODE. Lane `model_performance-6da` +ran `summary_call_mode: "fork"` end-to-end (S5-CRAC, n=3/arm balanced across +two containers, gpt-5.6-terra@medium) and found it COST-NEUTRAL: total run +cost -0.8%, i.e. noise. The mechanism visibly works -- a forked call reads a +median 85.7% of its own prompt from cache, a standalone one reads 0.0% -- +but it pays for itself exactly: + + forked call: $0.00077 / 1k own-prompt tokens, median prompt 26,620 tok + standalone call: $0.00350 / 1k own-prompt tokens, median prompt 5,129 tok + 4.5x cheaper per token x 5.2x more tokens ~= 1.0 + +The trade is strongly span-size dependent, and THAT is the lever: + + standalone, span > 30,000 tok: $0.1410 mean (n=17) + standalone, span <= 15,000 tok: $0.0151 mean (n=40) + forked, any span: $0.0265 mean (n=20) -- roughly FLAT + +A fork wins ~5x on the tail and loses on the median. So gate it. The +break-even is DERIVED, not chosen: a standalone call pays for the SPAN at +the uncached rate, a fork pays for the PREFIX at the cached rate, so a fork +is cheaper exactly when span/prefix > 0.00077/0.00350 = 0.22. + +Evidence: `.amplifier/evaluation/probes/6da-summary-fork/FINDINGS.md` §6 in +the openai-evals-team-ci repo. + +The tests below are written against the ways this can silently go wrong: + + 1. THE DEFAULT MUST NOT MOVE -- TWICE OVER. Not only must plain + `standalone` be untouched, `fork` WITHOUT an explicit ratio must also + be byte-identical to fork mode before this feature existed. A + predicate that quietly turns itself on is a behavior change wearing an + opt-in's clothes. Group A. + 2. A PREDICATE THAT NEVER FIRES IS A NO-OP THAT PASSES ITS TESTS. Group B + pins the boundary from BOTH sides against a ratio measured from the + fixture itself, so "fires above, does not fire below" is asserted at + the exact threshold rather than somewhere vaguely near it. + 3. A DECLINE IS NOT A FAILURE, AND MUST NOT LOOK LIKE ONE. `fork_declines` + and `fork_fallbacks` answer different questions -- "the predicate + worked" vs "the fork broke". Group C proves they never contaminate + each other, in either direction, and that ordering puts alignment + first so no span size can buy a misaligned fork. + 4. NOTHING ELSE MOVES. Group D proves a declined fork produces the + standalone request byte-for-byte, consumes no `_seq`, and leaves span + selection and tool-pair integrity exactly where the control leaves them. +""" + +import hashlib +import json +import logging + +import pytest +from amplifier_core import ChatResponse, Message, TextBlock +from amplifier_module_context_simple import ( + DEFAULT_FORK_MIN_SPAN_RATIO, + SimpleContextManager, + mount, +) + + +class _FakeProvider: + """Records every request handed to it so tests can assert the exact + shape that would go on the wire.""" + + def __init__(self, response_text: str = "SUMMARY TEXT"): + self.response_text = response_text + self.calls: list = [] + + async def complete(self, request): + self.calls.append(request) + return ChatResponse(content=[TextBlock(type="text", text=self.response_text)]) + + +class _Coordinator: + def __init__(self): + self.hooks = None + self.mounted = {} + + async def mount(self, kind, instance): + self.mounted[kind] = instance + + +def _tools(): + return [ + { + "name": "bash", + "description": "run a command", + "parameters": {"type": "object", "properties": {}}, + } + ] + + +def _digest(messages) -> str: + dumps = [] + for m in messages: + dumps.append(m.model_dump() if isinstance(m, Message) else Message(**m).model_dump()) + return hashlib.sha256( + json.dumps(dumps, sort_keys=True, default=str).encode() + ).hexdigest() + + +def _strip_timestamps(messages: list[dict]) -> list[dict]: + result = [] + for msg in messages: + meta = dict(msg.get("metadata") or {}) + meta.pop("timestamp", None) + result.append({**msg, "metadata": meta}) + return result + + +async def _await_pending_task(context: SimpleContextManager) -> None: + task = context._summarization_task + assert task is not None, "expected a background summarization task in flight" + await task + + +def _summary_manager(**overrides) -> SimpleContextManager: + kwargs = dict( + compaction_strategy="summary", + compaction_notice_enabled=False, + protected_recent=0.5, + summary_trigger=0.3, + target_usage=0.2, + compact_threshold=0.99, # keep the outer progressive gate CLOSED + max_tokens=1_000_000, + ) + kwargs.update(overrides) + return SimpleContextManager(**kwargs) + + +async def _fill(context: SimpleContextManager, turns: int = 30) -> None: + for i in range(turns): + await context.add_message({"role": "user", "content": f"user turn {i} " + "x" * 40}) + await context.add_message( + {"role": "assistant", "content": f"assistant reply {i} " + "y" * 40} + ) + + +async def _arm_below_trigger(context: SimpleContextManager) -> list[dict]: + """Serve one request with usage BELOW summary_trigger -- this is what + records the prefix a later fork appends to.""" + raw = context._estimate_tokens(context.messages) + context.max_tokens = int(raw / 0.1) + view = await context.get_messages_for_request(provider=_FakeProvider()) + assert context._is_summarizing is False, "must not trigger below summary_trigger" + return view + + +def _cross_trigger(context: SimpleContextManager) -> None: + raw = context._estimate_tokens(context.messages) + context.max_tokens = int(raw / 0.5) + + +async def _run_one_summarization(context: SimpleContextManager) -> _FakeProvider: + """Arm the fork seam, cross the trigger, and drain the background task.""" + await _fill(context) + context.note_request_sent(tools=_tools(), model="pinned-model") + await _arm_below_trigger(context) + _cross_trigger(context) + provider = _FakeProvider() + await context.get_messages_for_request(provider=provider) + await _await_pending_task(context) + return provider + + +async def _measure_realized_ratio(**overrides) -> float: + """Run one summarization with the predicate armed at a ratio nothing can + meet, purely to read back what span:prefix the fixture ACTUALLY produces. + + Every boundary test below is anchored to this measured number rather + than to a constant guessed by the test author -- otherwise "fires above + the threshold" degrades into "fires somewhere, probably". + """ + context = _summary_manager( + summary_call_mode="fork", summary_fork_min_span_ratio=99.0, **overrides + ) + await _run_one_summarization(context) + measure = context.last_summary_call_stats["span_measure"] + assert measure is not None and measure["span_ratio"] is not None + return measure["span_ratio"] + + +# --------------------------------------------------------------------------- +# Group A -- the default must not move, in EITHER mode +# --------------------------------------------------------------------------- + + +def test_default_ratio_is_none_in_every_default_configuration(): + assert SimpleContextManager().summary_fork_min_span_ratio is None + assert _summary_manager().summary_fork_min_span_ratio is None + assert _summary_manager(summary_call_mode="fork").summary_fork_min_span_ratio is None + + +def test_plain_fork_mode_resolves_to_no_predicate_at_all(): + """`fork` without an explicit ratio must keep forking unconditionally. + Turning the predicate on for it would silently change PR #27's shipped + behavior for anyone already using it.""" + assert _summary_manager(summary_call_mode="fork")._effective_fork_min_span_ratio() is None + assert _summary_manager()._effective_fork_min_span_ratio() is None + + +def test_auto_resolves_to_the_measured_break_even(): + context = _summary_manager(summary_call_mode="auto") + assert context.summary_call_mode == "auto" + assert context._effective_fork_min_span_ratio() == DEFAULT_FORK_MIN_SPAN_RATIO + + +def test_the_shipped_default_is_the_number_6da_measured(): + """0.22 is $0.00077/$0.00350 -- the cached rate over the uncached rate -- + from FINDINGS.md §6, not a round number someone liked. Pinned so a + future edit to the constant has to argue with this test.""" + assert DEFAULT_FORK_MIN_SPAN_RATIO == pytest.approx(0.00077 / 0.00350, abs=0.005) + + +def test_an_explicit_ratio_overrides_auto_in_both_directions(): + assert ( + _summary_manager( + summary_call_mode="auto", summary_fork_min_span_ratio=0.9 + )._effective_fork_min_span_ratio() + == 0.9 + ) + assert ( + _summary_manager( + summary_call_mode="fork", summary_fork_min_span_ratio=0.05 + )._effective_fork_min_span_ratio() + == 0.05 + ) + + +def test_auto_is_a_real_mode_and_never_warns(caplog): + with caplog.at_level(logging.WARNING): + context = SimpleContextManager(summary_call_mode="auto") + assert context.summary_call_mode == "auto" + assert not [r for r in caplog.records if "summary_call_mode" in r.message] + + +@pytest.mark.parametrize("bad", ["banana", -1.0, float("nan"), object()]) +def test_an_unusable_ratio_disables_the_predicate_loudly(caplog, bad): + with caplog.at_level(logging.WARNING): + context = SimpleContextManager(summary_fork_min_span_ratio=bad) + assert context.summary_fork_min_span_ratio is None + assert any("summary_fork_min_span_ratio" in r.message for r in caplog.records) + + +def test_zero_disables_the_predicate_silently(caplog): + """0 is how `compact_clear_at_least` already spells "off"; spelling it + the same way here must not be treated as a mistake.""" + with caplog.at_level(logging.WARNING): + context = SimpleContextManager(summary_fork_min_span_ratio=0) + assert context.summary_fork_min_span_ratio is None + assert not [r for r in caplog.records if "summary_fork_min_span_ratio" in r.message] + + +@pytest.mark.asyncio +async def test_standalone_mode_never_evaluates_the_predicate(): + """The default path must not so much as write a predicate attribute.""" + context = _summary_manager(summary_fork_min_span_ratio=0.5) + await _fill(context) + _cross_trigger(context) + await context.get_messages_for_request(provider=_FakeProvider()) + await _await_pending_task(context) + + assert context._last_fork_span_measure is None + assert context._summary_fork_declines == 0 + stats = context.last_summary_call_stats + assert stats["mode_used"] == "standalone" + assert stats["span_measure"] is None + assert stats["fork_declines"] == 0 + + +@pytest.mark.asyncio +async def test_fork_without_a_ratio_is_unchanged_from_before_the_predicate(): + """The regression that matters most: fork mode with the predicate off + must fork, and must not record a measurement it never took.""" + context = _summary_manager(summary_call_mode="fork") + provider = await _run_one_summarization(context) + + stats = context.last_summary_call_stats + assert stats["mode_used"] == "fork" + assert stats["reason"] is None + assert stats["span_measure"] is None, "predicate off must not measure" + assert stats["fork_declines"] == 0 + assert stats["fork_fallbacks"] == 0 + assert len(provider.calls[0].messages) > 2, "a fork, not a standalone pair" + + +@pytest.mark.asyncio +async def test_mount_threads_the_new_knobs_through(): + coordinator = _Coordinator() + await mount( + coordinator, + { + "compaction_strategy": "summary", + "summary_call_mode": "auto", + "summary_fork_min_span_ratio": 0.4, + }, + ) + context = coordinator.mounted["context"] + assert context.summary_call_mode == "auto" + assert context.summary_fork_min_span_ratio == 0.4 + + coordinator = _Coordinator() + await mount(coordinator, {"compaction_strategy": "summary", "summary_call_mode": "auto"}) + assert ( + coordinator.mounted["context"]._effective_fork_min_span_ratio() + == DEFAULT_FORK_MIN_SPAN_RATIO + ) + + coordinator = _Coordinator() + await mount(coordinator, {"compaction_strategy": "summary"}) + assert coordinator.mounted["context"].summary_fork_min_span_ratio is None + + +# --------------------------------------------------------------------------- +# Group B -- the predicate fires above the threshold and not below it +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_a_span_below_the_threshold_declines_the_fork(): + context = _summary_manager(summary_call_mode="fork", summary_fork_min_span_ratio=99.0) + provider = await _run_one_summarization(context) + + stats = context.last_summary_call_stats + assert stats["mode_used"] == "standalone" + assert "below the summary_fork_min_span_ratio" in stats["reason"] + assert stats["fork_declines"] == 1 + assert stats["fork_fallbacks"] == 0 + assert len(provider.calls[0].messages) == 2, "the standalone two-message pair" + + +@pytest.mark.asyncio +async def test_a_span_above_the_threshold_forks(): + context = _summary_manager( + summary_call_mode="fork", summary_fork_min_span_ratio=0.000_001 + ) + provider = await _run_one_summarization(context) + + stats = context.last_summary_call_stats + assert stats["mode_used"] == "fork" + assert stats["reason"] is None + assert stats["fork_declines"] == 0 + assert stats["span_measure"]["span_ratio"] > 0.000_001 + assert len(provider.calls[0].messages) > 2 + + +@pytest.mark.asyncio +async def test_the_threshold_is_an_inclusive_boundary_not_a_vague_region(): + """Pinned from both sides against the fixture's OWN realized ratio. + + A predicate tested only at 0.000001 and 99.0 would pass while being off + by an order of magnitude. These two assertions are separated by a single + epsilon around the real number. + """ + ratio = await _measure_realized_ratio() + + at_threshold = _summary_manager( + summary_call_mode="fork", summary_fork_min_span_ratio=ratio + ) + await _run_one_summarization(at_threshold) + assert at_threshold.last_summary_call_stats["mode_used"] == "fork", ( + "ratio == threshold must FORK: the comparison is >=, so the " + "break-even case takes the cache win rather than discarding it" + ) + + just_above = _summary_manager( + summary_call_mode="fork", summary_fork_min_span_ratio=ratio * 1.001 + ) + await _run_one_summarization(just_above) + assert just_above.last_summary_call_stats["mode_used"] == "standalone" + + +@pytest.mark.asyncio +async def test_auto_mode_actually_gates_on_the_default(): + """`auto` must be the predicate wired to DEFAULT_FORK_MIN_SPAN_RATIO -- + not a third name for unconditional forking.""" + ratio = await _measure_realized_ratio() + context = _summary_manager(summary_call_mode="auto") + await _run_one_summarization(context) + + stats = context.last_summary_call_stats + assert stats["mode_requested"] == "auto" + assert stats["span_measure"]["min_span_ratio"] == DEFAULT_FORK_MIN_SPAN_RATIO + expected = "fork" if ratio >= DEFAULT_FORK_MIN_SPAN_RATIO else "standalone" + assert stats["mode_used"] == expected + + +@pytest.mark.asyncio +async def test_the_measurement_is_reported_whichever_way_it_goes(): + """An eval arm has to be able to plot the realized span:prefix + distribution and re-derive its own threshold, including from the calls + the predicate let through.""" + for ratio, expected in ((99.0, "standalone"), (0.000_001, "fork")): + context = _summary_manager( + summary_call_mode="fork", summary_fork_min_span_ratio=ratio + ) + await _run_one_summarization(context) + measure = context.last_summary_call_stats["span_measure"] + assert context.last_summary_call_stats["mode_used"] == expected + assert measure["span_tokens"] > 0 + assert measure["prefix_tokens"] > 0 + assert measure["min_span_ratio"] == ratio + assert measure["span_ratio"] == pytest.approx( + measure["span_tokens"] / measure["prefix_tokens"] + ) + + +def test_the_default_classifies_6das_own_measured_populations_correctly(): + """The sharpest available check on the CONSTANT itself: replay the two + populations 6da actually measured through the shipped default and + require the verdict that matched the money. + + • the median pair (span 5,129 tok inside a 26,620 tok fork prompt, + ratio 0.193) is where the two arms cancelled -- it must DECLINE + • the tail (spans >30k, mean implied ~40k; standalone cost $0.1410 + against a flat ~$0.027 forked) must FORK + + A default that got either of these backwards would be shipping the + opposite of the finding. + """ + context = _summary_manager(summary_call_mode="auto") + threshold = context._effective_fork_min_span_ratio() + + median_ratio = 5_129 / 26_620 + assert ( + context._fork_span_declined_reason(5_129, 26_620, median_ratio, threshold) + is not None + ), "6da's median pair is BELOW break-even and must decline" + + tail_ratio = 40_000 / 60_000 + assert ( + context._fork_span_declined_reason(40_000, 60_000, tail_ratio, threshold) is None + ), "6da's >30k tail is where the fork wins ~5x and must fork" + + +def test_a_zero_token_prefix_does_not_divide_by_zero(): + """Degenerate input the ladder can hand us. No opinion is the right + answer -- a zero-token prefix has no cache worth protecting.""" + context = _summary_manager(summary_call_mode="auto") + span_tokens, prefix_tokens, ratio = context._fork_span_measure( + [{"role": "user", "content": "hi"}], [] + ) + assert prefix_tokens == 0 + assert ratio is None + assert ( + context._fork_span_declined_reason(span_tokens, prefix_tokens, None, 0.22) is None + ) + + +def test_a_ratio_above_one_is_legal_and_can_never_be_met(): + """The span is a SUBSET of the prefix, so >1.0 is a deliberate "arm the + plumbing, never fork" setting -- not an error to be clamped away.""" + context = _summary_manager(summary_call_mode="fork", summary_fork_min_span_ratio=2.0) + assert context._effective_fork_min_span_ratio() == 2.0 + assert context._fork_span_declined_reason(500, 1000, 0.5, 2.0) is not None + + +# --------------------------------------------------------------------------- +# Group C -- a decline is not a fallback, and alignment is checked first +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_a_decline_never_moves_the_fallback_counter(): + """`fork_fallbacks` is the signal an eval arm reads to detect SILENT + UNFORKING. A working predicate incrementing it would make the feature + indistinguishable from the defect.""" + context = _summary_manager(summary_call_mode="fork", summary_fork_min_span_ratio=99.0) + await _run_one_summarization(context) + assert context._summary_fork_declines == 1 + assert context._summary_fork_fallbacks == 0 + + +@pytest.mark.asyncio +async def test_a_misalignment_is_still_a_fallback_even_with_the_predicate_on(): + """The mirror image, and the ordering proof: no span size may buy a + misaligned fork. `note_request_sent()` is never called here, so the + fork is impossible for a reason that has nothing to do with cost -- and + that reason, not the predicate, must be the one reported.""" + context = _summary_manager( + summary_call_mode="fork", summary_fork_min_span_ratio=0.000_001 + ) + await _fill(context) + _cross_trigger(context) + await context.get_messages_for_request(provider=_FakeProvider()) + await _await_pending_task(context) + + stats = context.last_summary_call_stats + assert stats["mode_used"] == "standalone" + assert "note_request_sent() has never been called" in stats["reason"] + assert stats["fork_fallbacks"] == 1 + assert stats["fork_declines"] == 0 + assert stats["span_measure"] is None, ( + "the predicate must not even measure a fork that could not have " + "happened -- otherwise the recorded distribution is contaminated " + "with calls the threshold never governed" + ) + + +@pytest.mark.asyncio +async def test_a_decline_speaks_at_info_and_never_at_warning(caplog): + """A warning means something is wrong. The predicate declining is the + feature working, and must not train anyone to ignore fork warnings.""" + context = _summary_manager(summary_call_mode="fork", summary_fork_min_span_ratio=99.0) + with caplog.at_level(logging.DEBUG, logger="amplifier_module_context_simple"): + await _run_one_summarization(context) + + warnings = [r for r in caplog.records if r.levelno >= logging.WARNING] + assert not warnings, f"unexpected warning(s): {[r.message for r in warnings]}" + infos = [r for r in caplog.records if r.levelno == logging.INFO and "DECLINED" in r.message] + assert len(infos) == 1 + + +@pytest.mark.asyncio +async def test_repeated_declines_announce_once_but_still_count(): + """Every decline message carries its own token counts, so deduping on + the message would dedupe nothing and emit a line per summarization.""" + context = _summary_manager(summary_call_mode="fork", summary_fork_min_span_ratio=99.0) + await _run_one_summarization(context) + with caplog_at_info() as records: + # Drive two more declines directly -- distinct messages, same kind. + context._note_fork_declined("the span is 10 tok against a 900 tok prefix") + context._note_fork_declined("the span is 11 tok against a 950 tok prefix") + assert context._summary_fork_declines == 3, "every decline is counted" + assert not [r for r in records if "DECLINED" in r.message], "announced only once" + + +class caplog_at_info: + """Minimal record collector -- pytest's caplog fixture cannot be + re-entered inside a test that already used it in a helper.""" + + def __enter__(self): + self.records: list[logging.LogRecord] = [] + self.handler = logging.Handler() + self.handler.emit = lambda record: self.records.append(record) + self.logger = logging.getLogger("amplifier_module_context_simple") + self.previous = self.logger.level + self.logger.setLevel(logging.INFO) + self.logger.addHandler(self.handler) + return self.records + + def __exit__(self, *exc): + self.logger.removeHandler(self.handler) + self.logger.setLevel(self.previous) + return False + + +# --------------------------------------------------------------------------- +# Group D -- nothing else moves +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_a_declined_fork_sends_the_standalone_request_byte_for_byte(): + """The whole safety argument: a decline is not a third request shape. + Asserted against a control manager that was never in fork mode at all.""" + declined = _summary_manager(summary_call_mode="fork", summary_fork_min_span_ratio=99.0) + control = _summary_manager() + + declined_provider = await _run_one_summarization(declined) + + await _fill(control) + await _arm_below_trigger(control) + _cross_trigger(control) + control_provider = _FakeProvider() + await control.get_messages_for_request(provider=control_provider) + await _await_pending_task(control) + + assert _digest(declined_provider.calls[0].messages) == _digest( + control_provider.calls[0].messages + ) + assert declined_provider.calls[0].model == control_provider.calls[0].model is None + assert declined_provider.calls[0].tools == control_provider.calls[0].tools is None + + +@pytest.mark.asyncio +async def test_a_declined_fork_still_produces_the_summary(): + """Declining must cost the summary nothing. A predicate that saves money + by losing summaries is not a cost win.""" + context = _summary_manager(summary_call_mode="fork", summary_fork_min_span_ratio=99.0) + await _run_one_summarization(context) + assert context._pending_summary is not None + assert context._summarization_failures == 0 + + +@pytest.mark.asyncio +async def test_the_predicate_consumes_no_seq_and_does_not_touch_history(): + context = _summary_manager(summary_call_mode="auto") + await _fill(context) + context.note_request_sent(tools=_tools(), model="pinned-model") + await _arm_below_trigger(context) + before = json.dumps(_strip_timestamps(context.messages), default=str) + seq_before = context._next_seq + + _cross_trigger(context) + await context.get_messages_for_request(provider=_FakeProvider()) + await _await_pending_task(context) + + # The summarizer call has completed; the swap-in happens on the NEXT + # served view. So at this instant nothing may have moved at all. + assert context._next_seq == seq_before, "the predicate must consume no _seq" + assert json.dumps(_strip_timestamps(context.messages), default=str) == before + + +@pytest.mark.asyncio +async def test_the_predicate_does_not_change_which_span_is_selected(): + """The predicate changes how the summarizer is CALLED, never what is + selected -- including the tool-pair boundary snapping.""" + gated = _summary_manager(summary_call_mode="fork", summary_fork_min_span_ratio=99.0) + control = _summary_manager() + for ctx in (gated, control): + for i in range(12): + await ctx.add_message({"role": "user", "content": f"do thing {i} " + "x" * 30}) + await ctx.add_message( + { + "role": "assistant", + "content": "working", + "tool_calls": [{"id": f"call-{i}", "tool": "bash", "arguments": {}}], + } + ) + await ctx.add_message( + {"role": "tool", "tool_call_id": f"call-{i}", "content": "out " + "y" * 30} + ) + + gated.note_request_sent(tools=_tools(), model="pinned-model") + await _arm_below_trigger(gated) + await _arm_below_trigger(control) + _cross_trigger(gated) + _cross_trigger(control) + + for ctx in (gated, control): + await ctx.get_messages_for_request(provider=_FakeProvider()) + await _await_pending_task(ctx) + + assert gated.last_summary_call_stats["mode_used"] == "standalone", "predicate declined" + assert sorted(gated._pending_summary["seqs"]) == sorted(control._pending_summary["seqs"]) + + +@pytest.mark.asyncio +async def test_a_declined_fork_leaves_the_next_served_view_identical(): + gated = _summary_manager(summary_call_mode="fork", summary_fork_min_span_ratio=99.0) + control = _summary_manager() + + await _run_one_summarization(gated) + await _fill(control) + await _arm_below_trigger(control) + _cross_trigger(control) + await control.get_messages_for_request(provider=_FakeProvider()) + await _await_pending_task(control) + + gated_view = await gated.get_messages_for_request(provider=_FakeProvider()) + control_view = await control.get_messages_for_request(provider=_FakeProvider()) + assert _digest(_strip_timestamps(gated_view)) == _digest(_strip_timestamps(control_view))