From a72e49aef117eed908f7faf3912a8e3409368647 Mon Sep 17 00:00:00 2001 From: bkrabach <702425+bkrabach@users.noreply.github.com> Date: Wed, 2 Sep 2026 17:07:34 -0700 Subject: [PATCH 1/7] Revert "fix: _capture_fork_prefix no longer forks onto an array that was never sent (#jnt) (#28)" This reverts commit 5cdbc62253ca80cb93f11a2fa6f7a5d3940394c78. Not in the owner's literal revert list (#20-27), but merged AFTER #27 and exists solely to fix a bug in the summary_call_mode="fork" code path that #27 introduced. Once #27 (the fork feature itself) is reverted next, this fix has no surviving target -- keeping it would leave dead code / an orphaned test file referencing a feature no longer on main. Reverting it first, before #27, keeps the revert of #27 itself clean. --- DONE-NOTE.md | 174 --------------- amplifier_module_context_simple/__init__.py | 138 ++---------- tests/test_summary_call_mode_fork.py | 229 +------------------- 3 files changed, 30 insertions(+), 511 deletions(-) diff --git a/DONE-NOTE.md b/DONE-NOTE.md index c790066..0bd7671 100644 --- a/DONE-NOTE.md +++ b/DONE-NOTE.md @@ -1129,177 +1129,3 @@ would bury this change in unrelated noise. No PII, no team-internal data, no individual attribution. No merges to main. No files touched outside this module. No infrastructure created; nothing to tear down. - ---- - -# DONE-NOTE - model_performance-jnt - -**`_capture_fork_prefix()` appends to an array that was never sent (11 of 20 -forks measured).** - -**Verdict: CORRUPTING, not cosmetic.** It is not dead code and it produces a -record. It silently substitutes a message array that never went on the wire -for one that did, on a path whose entire purpose is byte-parity with the wire. -Fixed. **No 6da measured number is invalidated** — see §3. - -## 1. The array, named, at file:line - -Everything below is `amplifier_module_context_simple/__init__.py` at the -pre-fix commit `a877b36`. - -| | | -|---|---| -| **the array** | **`self._last_request_view`** | -| created | **:1015** (`__init__`), written **:1778** inside `_finalize_view()` | -| what it holds | this module's own last RETURNED view, pre-strip | -| why it is "never sent" | it is written on **every view served**, and a view served is not a request sent | -| consumed by | **:4187** `_capture_fork_prefix()`, as the fallback source | -| flows to | `_maybe_trigger_summary_compaction` **:4040** → `_run_summary_compaction_task` → `_build_fork_request` **:4422** → the provider | - -The selection at **:4187–4198** preferred the caller's recorded wire array -(`_sent_messages`, **:1017/:1210**) only while -`_sent_serial == _view_serial` (**:4189**), and substituted -`_last_request_view` whenever that equality failed (**:4187**, **:4192–4197** -— `logger.debug`, not warning). - -**Why the equality fails in production.** `_view_serial` counts **views -served**, not **requests sent**, and the two are not 1:1. The real -orchestrator serves the view more than once per sent request: -`amplifier-module-loop-streaming/__init__.py` calls -`context.get_messages_for_request()` at **:3215** and then re-fetches at -**:3329** and **:3453** after persisting an ephemeral injection — up to three -views for one `ChatRequest`. The summary trigger is evaluated inside *every* -one of them (**:1364**, which runs *before* that call's `_finalize_view` at -**:1480/:1482**). So: - -- trigger on the **first** view of a request → serials match → the wire array - is used → **correct fork** (wire offset 0 or 1); -- trigger on a **re-fetched** view → serials differ by 1–2 → the wire array is - discarded in favour of `_last_request_view`, which at that instant holds - **the view the re-fetch just superseded** — built, thrown away, never sent. - -That is the 9-vs-11 split 6da measured, and it reproduces exactly. - -## 2. Consequence — unambiguous - -**Corrupting to fork-mode behaviour, and self-concealing.** - -1. **Guaranteed cache miss.** A superseded view is not a prefix any provider - holds. Fork mode's only justification is appending onto a cached prefix; a - fork that misses pays full price for the whole conversation, which is - *strictly worse* than the standalone call it replaces. -2. **Silent.** `last_summary_call_stats["mode_used"]` reported `"fork"` for - all 20 calls. The substitution was `logger.debug`. There was no field - distinguishing the two sources — this is precisely why 6da had to - reconstruct the distinction from the provider's request log. -3. **Backwards under uncertainty.** The check traded the array with *positive - evidence* of having been sent (the caller said so) for one with *none* - (fate unknown to this module), and did so exactly when uncertainty was - highest. - -Not affected: history, `_seq` allocation, span selection, tool-pair -integrity, the served view, or any default-mode behaviour. The blast radius -is fork mode's cache economics and the honesty of its self-report. - -## 3. Does this invalidate any of 6da's measured numbers? **NO.** - -Stated plainly for the manager, because the item asked for it loudly: - -- **G-FORK-PREFIX (2/7/11 offset distribution, 45% aligned) — VALID, and is - the direct measurement of this bug.** Scored from the wire, not from the - module's self-report. -- **G-FORK-CACHED, G-FORK-NOBOUNDARY, the Anthropic guardrail, quality - parity, and the −0.8% run-cost delta — VALID.** All are wire/usage-derived - and none depend on `_capture_fork_prefix()` having chosen correctly. -- **The §7 correction (summarizer share ≈30%, not 2.4%/8.3–10.9%) — VALID - and untouched.** Independent of the fork path. - -**One caveat, in 6da's favour, not against it:** the −0.8% run-cost delta was -measured with only ~45% of forks byte-aligned. It is a **lower bound** on -what a correctly-aligned fork arm would deliver, not an upper bound. 6da's -"the mechanism works and the lever does not pay / DON'T-SHIP as-is" verdict -therefore **still stands as written**, but its cost figure is now known to -have been measured on a partially-broken treatment and should be **re-measured -before the DON'T-SHIP call is made final**. 6da itself flagged this -("one of them has a cheap fix worth a follow-up item"); this is that fix. - -## 4. The fix (minimal) - -`_capture_fork_prefix()` (**:4176**) no longer substitutes: - -- a recorded wire array, when one exists, is **used** — it is the only source - carrying positive evidence it was on the wire, so it is never traded for one - that carries none. Extra views served since the send are **reported, not - acted on**; -- the module's own view is used **only** when the caller has never supplied a - message array (the documented explicit-breakpoint/Anthropic path, unchanged); -- staleness in the wire record is still caught, but by an **exact** check - instead of a proxy: a record too old to contain the span fails - `_prefix_contains_span` and **refuses LOUDLY** — standalone call, `WARNING`, - `_summary_fork_fallbacks` incremented, named `reason`. "A fork that silently - missed" is no longer reachable on this path. - -Also added, because 6da needed it and could not get it: **`prefix_source`** -(`"wire_record"` / `"module_view"` / `None`) and **`prefix_views_since_send`** -on `last_summary_call_stats`. The next arm can separate the two populations -from the module's own report instead of reconstructing them from the wire. - -**Config surface: unchanged. Default `summary_call_mode` remains -`"standalone"`.** - -## 5. Tests - -`286 passed, 1 skipped` (was 281 passed at `a877b36`). `ruff check`: clean. - -New Group F in `tests/test_summary_call_mode_fork.py`: - -| Test | Pins | -|---|---| -| `test_a_re_fetched_view_does_not_displace_the_recorded_wire_array` | THE regression: a superseding re-fetch must not displace the wire array | -| `test_the_module_view_is_never_substituted_when_a_wire_record_exists` | same defect from the other side: never-sent content cannot reach the fork | -| `test_prefix_source_names_the_module_view_path_honestly` | the module-view path is allowed but reported as what it is | -| `test_prefix_source_is_none_when_the_call_did_not_fork` | a refused fork claims no alignment | -| `test_tool_pair_integrity_and_seq_stability_survive_the_re_fetch_path` | no `_seq` consumed, history byte-identical, same span absorbed, served view identical to an unforked control | - -**These three fail against the pre-fix selection logic and pass against the -fix** — verified by temporarily restoring the old branch and re-running; they -are load-bearing, not decoration. - -One existing test changed: `test_a_stale_caller_message_record_is_ignored_not_trusted` -→ `test_a_stale_caller_message_record_refuses_loudly_not_silently`. It -asserted the substitution *as correct behaviour*; it now asserts the loud -refusal. The rewritten docstring records why the original resolution was -wrong, so the reversal is not silent. - -Default-mode byte-identity re-verified by the pre-existing Group A/C tests, -strengthened with `assert context._fork_prefix_source is None` in -`test_default_mode_never_records_a_fork_prefix`. - -## 6. Residual, disclosed - -The **module-view path** (`note_request_sent(tools=...)` with no `messages`) -can still append to a superseded view — this module genuinely cannot know -whether its own view was sent. Not silently, now: `prefix_source == -"module_view"` says so on every call. **A caller that wants byte-parity must -pass `messages`.** Closing this properly needs a caller-side confirmation -signal, which is an orchestrator change and out of this lane's scope. - -Unchanged and still true: fork mode cannot fork the **first** summarization of -a CLI turn (each turn is a fresh `amplifier run --resume` process, and the -trigger is evaluated before any request is sent). 6da measured 12 of 24 -refusals from this; this fix does not address it. - -## 7. Deliverable ledger - -| Deliverable | Status | -|---|---| -| DRAFT PR on origin, branch `lane/jnt-fork-prefix-capture`, tests green, default inline byte-identical | **DONE** | -| The array named at file:line with why it was never sent + cosmetic-vs-corrupting verdict | **DONE** — §1, §2 (**corrupting**) | -| Explicit statement of whether any of 6da's measured fork numbers are invalidated | **DONE** — §3 (**none invalidated**; −0.8% is a lower bound and warrants re-measurement) | -| DONE-NOTE.md in the PR body | **DONE** — this section | - -**Spend: $0.00.** No API calls, no DTU, no containers, no infrastructure -created — the item was answerable from the code, the shipped tests, and 6da's -existing evidence files. Nothing to tear down; nothing registered in the infra -ledger. No PII or team-internal data. No merge to main. No files touched -outside this module. diff --git a/amplifier_module_context_simple/__init__.py b/amplifier_module_context_simple/__init__.py index 47f8055..9fed112 100644 --- a/amplifier_module_context_simple/__init__.py +++ b/amplifier_module_context_simple/__init__.py @@ -360,24 +360,6 @@ # "inline" is accepted as an alias for "standalone": the lane brief that # commissioned this work named the default mode "inline" while the work # item named it "standalone". Both mean "today's behavior, unchanged". -# Where a forked summarizer call's prefix came from, reported verbatim on -# `last_summary_call_stats["prefix_source"]`. This distinction is not -# cosmetic: only ONE of these two sources carries positive evidence that -# the array was ever on the wire. -# -# "wire_record" -- the caller's own note_request_sent(messages=...) -# record. The caller told us it sent exactly this. -# "module_view" -- this module's last RETURNED view. Whether it was -# actually sent is unknown to this module: an -# orchestrator may append a tail to it, or discard it -# entirely and re-fetch a fresh view before sending -# (amplifier's loop-streaming does exactly that, 1-3 -# times per sent request). A superseded view was never -# on the wire, so forking onto one is a guaranteed -# cache miss wearing a correct-looking API call. -FORK_PREFIX_SOURCE_WIRE = "wire_record" -FORK_PREFIX_SOURCE_VIEW = "module_view" - SUMMARY_CALL_MODE_STANDALONE = "standalone" SUMMARY_CALL_MODE_FORK = "fork" _VALID_SUMMARY_CALL_MODES = (SUMMARY_CALL_MODE_STANDALONE, SUMMARY_CALL_MODE_FORK) @@ -1037,11 +1019,6 @@ def __init__( self._sent_tools: Any = None self._sent_tools_supplied: bool = False self._sent_model: str | None = None - # Which of the two sources the last captured fork prefix came from - # (FORK_PREFIX_SOURCE_*). Reported on `last_summary_call_stats` so a - # measurement can tell a wire-parity fork from a module-view fork - # WITHOUT reconstructing it from the provider's request log. - self._fork_prefix_source: str | None = None # Observability: what the LAST summarizer call actually did, and how # many times a requested fork had to fall back. An eval arm reads # these to tell a real fork from a silently unforked one. @@ -1203,19 +1180,12 @@ def note_request_sent( enters `self.messages`, consumes a `_seq`, or moves any compaction state. Callers that do not know about it lose nothing. - Call it EVERY request, not once. The most recent `messages` record - is the only array this module has positive evidence was ever on the - wire, so it is what a fork appends to -- a one-time wiring would - have turn 40's fork append to turn 1's request. That case is not - silently rerouted (rerouting is what produced the never-sent-array - bug); it is caught exactly, by the span-presence check, and refused - LOUDLY as a standalone call with a named reason. - - Calling it more than once per request is harmless, and so is - serving the view more than once per request: extra views do not - invalidate the record. `last_summary_call_stats` reports - `prefix_views_since_send` so that re-fetching is observable rather - than inferred. + Call it EVERY request, not once. A `messages` record is only used + while it still describes the most recent request this module served + -- a one-time wiring would otherwise have turn 40's fork append to + turn 1's request, which is a guaranteed cache miss dressed up as a + correct API call. A stale record is ignored (this module's own last + view is used instead), never trusted. Args: messages: The exact message array sent, if known. Gives the fork @@ -1250,11 +1220,9 @@ def last_summary_call_stats(self) -> dict[str, Any] | None: None before the first one. Otherwise a dict with `mode_requested`, `mode_used`, `reason` (None when the requested mode was honored), - `prefix_messages`, `prefix_source` (FORK_PREFIX_SOURCE_*, None when - not forked), `prefix_views_since_send`, and `fork_fallbacks` - (session-cumulative). This is how an eval arm distinguishes a real - fork from a silently unforked one -- and a wire-parity fork from a - module-view one -- WITHOUT patching the module. + `prefix_messages`, and `fork_fallbacks` (session-cumulative). This + is how an eval arm distinguishes a real fork from a silently + unforked one WITHOUT patching the module. """ return dict(self._last_summary_call) if self._last_summary_call else None @@ -1662,7 +1630,6 @@ def _reset_summary_strategy_state(self) -> None: self._sent_tools = None self._sent_tools_supplied = False self._sent_model = None - self._fork_prefix_source = None async def should_compact(self) -> bool: """Check if context should be compacted. @@ -4209,68 +4176,26 @@ def _fork_armed(self) -> bool: def _capture_fork_prefix(self) -> list[dict[str, Any]] | None: """Snapshot the message array a forked call would append to. - Uses the caller's own `note_request_sent(messages=...)` record when - there is one -- byte-parity with the wire, including any tail the - orchestrator injected after this module returned. Only when the - caller has never supplied a message array does this fall back to - the module's own last returned view, which is where an + Prefers the caller's own `note_request_sent(messages=...)` record + (byte-parity with the wire, including any tail the orchestrator + injected after this module returned). Falls back to this module's + last returned view, which still ends exactly where an explicit-breakpoint provider places its cache breakpoint. - - WHY THERE IS NO SILENT DOWNGRADE BETWEEN THE TWO (the bug this - replaced): the previous implementation preferred the wire record - only while `_sent_serial == _view_serial`, and substituted - `_last_request_view` whenever that equality failed. That equality - asks "have any views been served since the caller last confirmed a - send?" -- which conflates two situations a view counter cannot tell - apart, and gets the important one backwards. - - A real orchestrator serves the view MORE THAN ONCE per sent request - (amplifier's loop-streaming re-fetches after persisting an ephemeral - injection -- `get_messages_for_request()` at three separate call - sites in one iteration). The summary trigger is evaluated inside - EVERY one of those calls. On the second and third, the serials no - longer match, and the old code swapped the caller's genuine wire - array for `_last_request_view` -- which at that instant holds a view - that was built, superseded by the re-fetch, and NEVER SENT. - - Measured on the wire (model_performance-6da, 20 forked calls): - 9 appended to a request the provider actually saw; the other 11 - appended to an array that was never sent as any request. The - substitution was invisible from outside -- `mode_used` reported - "fork" either way. - - The asymmetry that settles it: the wire record is the only source - carrying positive evidence that it was ever on the wire, so it is - never traded for one that carries none. Staleness in the wire record - is still caught, but by an EXACT check rather than a proxy: a record - too old to contain the span being absorbed fails - `_prefix_contains_span` and refuses LOUDLY (standalone + warning + - counter), which is the outcome a fork that cannot be byte-aligned is - supposed to have. """ if not self._fork_armed(): return None + source = self._last_request_view if self._sent_messages is not None: - self._fork_prefix_source = FORK_PREFIX_SOURCE_WIRE - if self._sent_serial != self._view_serial: - # Not an error, and deliberately not a substitution: the - # caller has served extra views since confirming this send - # (a re-fetch, or a view that was never sent at all). The - # last CONFIRMED array is still the best-evidenced prefix. + if self._sent_serial == self._view_serial: + source = self._sent_messages + else: logger.debug( - "context-simple: fork prefix is the caller's recorded " - f"wire array from view {self._sent_serial} (now at view " - f"{self._view_serial}); {self._view_serial - (self._sent_serial or 0)} " - "view(s) have been served since it was confirmed sent, " - "which is normal for an orchestrator that re-fetches the " - "view within a single request" + "context-simple: ignoring a stale note_request_sent() " + f"message record (recorded at view {self._sent_serial}, " + f"now at view {self._view_serial}); appending to this " + "module's own last returned view instead" ) - return list(self._sent_messages) - if self._last_request_view is not None: - self._fork_prefix_source = FORK_PREFIX_SOURCE_VIEW - return list(self._last_request_view) - self._fork_prefix_source = None - return None + return list(source) if source is not None else None @staticmethod def _message_identity(msg: dict[str, Any]) -> tuple[str, str, str]: @@ -4549,25 +4474,6 @@ async def _run_summary_compaction_task( if call_mode == SUMMARY_CALL_MODE_FORK else 0 ), - # WHICH array the fork appended to, from the module's own - # report rather than reconstructed from the wire. Only one - # of the two sources is evidenced as having been sent (see - # FORK_PREFIX_SOURCE_*), so a measurement that cannot see - # this field cannot tell a byte-aligned fork from a fork - # onto a view the provider never received. - "prefix_source": ( - self._fork_prefix_source - if call_mode == SUMMARY_CALL_MODE_FORK - else None - ), - # How many views were served since the caller last confirmed - # a send. >0 is normal (an orchestrator may re-fetch the view - # within one request); it is reported, not acted on. - "prefix_views_since_send": ( - self._view_serial - self._sent_serial - if self._sent_serial is not None - else None - ), "fork_fallbacks": self._summary_fork_fallbacks, } diff --git a/tests/test_summary_call_mode_fork.py b/tests/test_summary_call_mode_fork.py index c75af84..e039f93 100644 --- a/tests/test_summary_call_mode_fork.py +++ b/tests/test_summary_call_mode_fork.py @@ -242,7 +242,6 @@ async def test_default_mode_never_records_a_fork_prefix(): assert context._last_request_view is None assert context._sent_tools_supplied is False assert context._summary_fork_fallbacks == 0 - assert context._fork_prefix_source is None assert context.last_summary_call_stats is None @@ -608,24 +607,11 @@ async def test_fork_refuses_when_the_prefix_ends_on_unanswered_tool_calls(): @pytest.mark.asyncio -async def test_a_stale_caller_message_record_refuses_loudly_not_silently(caplog): +async def test_a_stale_caller_message_record_is_ignored_not_trusted(): """A caller that wires note_request_sent() ONCE (startup helper, first - turn only) would have turn N's fork append to turn 1's request. - - This test previously asserted the OPPOSITE resolution -- that the - module silently substituted its own last returned view. That - substitution is the defect this file now pins against - (model_performance-jnt): the module cannot know whether its own view - was ever sent, and under a real orchestrator that re-fetches the view - within a single request, the view it substitutes is one that was - superseded and never went on the wire. Trading an array KNOWN to have - been sent for one whose fate is unknown is backwards, and it was - invisible -- `mode_used` said "fork" either way. - - The correct resolution for a record too old to be usable is the one - every other misalignment already gets: refuse, run standalone, and SAY - SO. Caught exactly (the span is not in that prefix), not by a proxy. - """ + turn only) would otherwise have turn N's fork append to turn 1's + request -- a guaranteed miss AND a wasted cache write, wearing a + correct-looking API call. The fresh module view must win.""" context = _summary_manager(summary_call_mode="fork") await _fill(context, turns=10) # Turn 1: the caller records what it sent. @@ -639,28 +625,13 @@ async def test_a_stale_caller_message_record_refuses_loudly_not_silently(caplog) _cross_trigger(context) provider = _FakeProvider() - with caplog.at_level(logging.WARNING): - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - stats = context.last_summary_call_stats - assert stats["mode_used"] == "standalone", ( - "a prefix too stale to contain the span must refuse, not fork onto " - "a substituted view" - ) - assert "not present in the recorded prefix" in stats["reason"] - assert context._summary_fork_fallbacks == 1 - assert any("ran STANDALONE instead" in r.message for r in caplog.records) + await context.get_messages_for_request(provider=provider) + await _await_pending_task(context) - # The substitution specifically must not have happened: the standalone - # request is the two-message one, not an append onto the fresh view. request = provider.calls[0] - assert len(request.messages) == 2 - assert _digest(request.messages) != _digest(fresh_view) + assert context.last_summary_call_stats["mode_used"] == "fork" + assert _digest(request.messages[:-1]) == _digest(fresh_view) assert not any("long ago" in str(m.content) for m in request.messages) - # And the summary still happened -- refusing costs today's price, never - # the summary itself. - assert context._pending_summary is not None @pytest.mark.asyncio @@ -902,187 +873,3 @@ async def test_concurrent_forks_are_still_serialized_by_the_in_flight_guard(): assert len(provider.calls) == 1 assert context._pending_summary is not None assert context.last_summary_call_stats["mode_used"] == "fork" - - -# --------------------------------------------------------------------------- -# Group F -- the fork prefix is an array that was actually SENT -# -# model_performance-jnt. Measured on the wire (model_performance-6da, 20 -# forked calls): 9 appended to a request the provider actually saw, and 11 -# appended to an array that was never sent as any request. The module -# reported `mode_used == "fork"` for all 20. -# -# Cause: `_capture_fork_prefix()` preferred the caller's recorded wire array -# only while `_sent_serial == _view_serial`, and substituted -# `_last_request_view` otherwise. A real orchestrator serves the view more -# than once per sent request (amplifier's loop-streaming re-fetches after -# persisting an ephemeral injection), the trigger is evaluated inside every -# one of those calls, and on the second the substituted view is one that was -# superseded by the re-fetch and never went on the wire. -# -# These tests are written against that substitution, not against the happy -# path -- which the existing Group B tests already cover. -# --------------------------------------------------------------------------- - - -async def _serve_below_trigger(context: SimpleContextManager) -> list[dict]: - """Serve one more view without re-arming, i.e. a re-fetch within the - same request. `_arm_below_trigger` is the first view of a request; this - is the second one, which the orchestrator then supersedes.""" - view = await context.get_messages_for_request(provider=_FakeProvider()) - assert context._is_summarizing is False - return view - - -@pytest.mark.asyncio -async def test_a_re_fetched_view_does_not_displace_the_recorded_wire_array(): - """THE REGRESSION. An extra view served since the caller confirmed its - send is normal, not a reason to stop trusting the send. The wire array - is the only one with positive evidence of having been on the wire; the - intervening view has none.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - module_view = await _arm_below_trigger(context) - wire = [ - *module_view, - {"role": "user", "content": "injected"}, - ] - context.note_request_sent(wire, tools=_tools()) - sent_at = context._view_serial - - # The orchestrator re-fetches the view within the same request. This - # view is built, superseded, and never sent. - superseded = await _serve_below_trigger(context) - assert context._view_serial > sent_at, "the re-fetch must advance the view serial" - - _cross_trigger(context) - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - stats = context.last_summary_call_stats - assert stats["mode_used"] == "fork" - assert stats["prefix_source"] == "wire_record" - assert stats["prefix_views_since_send"] >= 1, ( - "the re-fetch must be visible in the stats, not silently acted on" - ) - - request = provider.calls[0] - assert _digest(request.messages[:-1]) == _digest(wire), ( - "the fork must append to the array the caller said it sent" - ) - assert _digest(request.messages[:-1]) != _digest(superseded) - assert "injected" in request.messages[-2].content - - -@pytest.mark.asyncio -async def test_the_module_view_is_never_substituted_when_a_wire_record_exists(): - """Belt and braces on the same defect, asserted from the other side: a - `_last_request_view` holding content that was demonstrably never sent - must not be able to reach the forked request at all.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - module_view = await _arm_below_trigger(context) - context.note_request_sent(module_view, tools=_tools()) - # A view that was built and discarded -- exactly what a re-fetch leaves - # behind, and what the old code would have forked onto. - context._last_request_view = [ - {"role": "user", "content": "a view that was never sent"} - ] - context._view_serial += 1 - - _cross_trigger(context) - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - request = provider.calls[0] - assert context.last_summary_call_stats["mode_used"] == "fork" - assert context.last_summary_call_stats["prefix_source"] == "wire_record" - assert not any("never sent" in str(m.content) for m in request.messages) - assert _digest(request.messages[:-1]) == _digest(module_view) - - -@pytest.mark.asyncio -async def test_prefix_source_names_the_module_view_path_honestly(): - """When the caller supplies tools but never a message array, the fork - appends to this module's own view -- whose send this module cannot - confirm. That is still allowed (it is what an explicit-breakpoint - provider wants), but it must be REPORTED as what it is, so a - measurement can separate the two populations without patching.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools()) - parent_view = await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - stats = context.last_summary_call_stats - assert stats["mode_used"] == "fork" - assert stats["prefix_source"] == "module_view" - assert stats["prefix_views_since_send"] is None, ( - "no message array was ever recorded, so there is no send to count from" - ) - assert _digest(provider.calls[0].messages[:-1]) == _digest(parent_view) - - -@pytest.mark.asyncio -async def test_prefix_source_is_none_when_the_call_did_not_fork(): - """A refused fork reports no prefix source. Reporting one would make a - standalone call look byte-aligned with something.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - stats = context.last_summary_call_stats - assert stats["mode_used"] == "standalone" - assert stats["prefix_source"] is None - - -@pytest.mark.asyncio -async def test_tool_pair_integrity_and_seq_stability_survive_the_re_fetch_path(): - """The re-fetch path must change WHICH array is appended to and nothing - else: the same span is selected, no `_seq` is consumed, history is - untouched, and the next served view matches an unforked control.""" - forked = _summary_manager(summary_call_mode="fork") - control = _summary_manager() - for ctx in (forked, control): - await _fill(ctx) - - view = await _arm_below_trigger(forked) - forked.note_request_sent([*view, {"role": "user", "content": "tail"}], tools=_tools()) - await _serve_below_trigger(forked) # the superseding re-fetch - await _arm_below_trigger(control) - for ctx in (forked, control): - _cross_trigger(ctx) - - seq_before = forked._next_seq - history_before = json.dumps(_strip_timestamps(forked.messages), default=str) - - for ctx in (forked, control): - await ctx.get_messages_for_request(provider=_FakeProvider("SAME SUMMARY")) - await _await_pending_task(ctx) - - assert forked.last_summary_call_stats["mode_used"] == "fork" - assert forked.last_summary_call_stats["prefix_source"] == "wire_record" - assert forked._next_seq == seq_before, "the fork must not consume a _seq" - assert ( - json.dumps(_strip_timestamps(forked.messages), default=str) == history_before - ), "the fork must not append to, reorder, or edit history" - assert set(forked._pending_summary["seqs"]) == set(control._pending_summary["seqs"]), ( - "the call mode must not change WHICH span is absorbed" - ) - - for ctx in (forked, control): - ctx.compact_threshold = 0.3 - forked_view = await forked.get_messages_for_request(provider=_FakeProvider()) - control_view = await control.get_messages_for_request(provider=_FakeProvider()) - assert _strip_timestamps(forked_view) == _strip_timestamps(control_view) - assert forked._removed_seqs == control._removed_seqs From d157004324b077ab3965ac6a5c8f0d67192dbf70 Mon Sep 17 00:00:00 2001 From: bkrabach <702425+bkrabach@users.noreply.github.com> Date: Wed, 2 Sep 2026 17:07:40 -0700 Subject: [PATCH 2/7] Revert "feat: summary_call_mode -- cache-safe fork of the summarization call (#27)" This reverts commit a877b36671e04add7d472c93240a606540533417. Merge policy: main carries wins only. summary_call_mode's fork mode was shipped with benefit explicitly labeled unmeasured (see the original merge note: "benefit correctly labeled unmeasured"). Unproven default-off feature -- belongs on a branch for evaluation, not on main. --- DONE-NOTE.md | 217 ----- README.md | 122 --- amplifier_module_context_simple/__init__.py | 594 +------------ tests/test_summary_call_mode_fork.py | 875 -------------------- 4 files changed, 16 insertions(+), 1792 deletions(-) delete mode 100644 tests/test_summary_call_mode_fork.py diff --git a/DONE-NOTE.md b/DONE-NOTE.md index 0bd7671..412d612 100644 --- a/DONE-NOTE.md +++ b/DONE-NOTE.md @@ -12,7 +12,6 @@ the other. Each note keeps its own `# DONE-NOTE - ` heading verbatim. | `model_performance-x7p` | `protected_tool_results=0` protected ALL tool results (negative-slice bug) | | `model_performance-x1r` | tool-result budget (token-denominated, head+tail, per-tool) + spill-to-disk | | `model_performance-2o9` | `clear_at_least` — a worth-the-rebuild predicate in front of compaction (+ summary shrink guard) | -| `model_performance-7k2` | `summary_call_mode` — cache-safe forking of the summarization call | --- @@ -913,219 +912,3 @@ property of the module, not new exposure, but now with one more field. No PII, no team-internal data, no individual attribution in any output. No merges to main; no files touched outside this module. - ---- - -# DONE-NOTE — W3-2 / `model_performance-7k2` - -`context-simple: cache-safe forking of the summarization call (summary_call_mode: "fork")` - -**Status: draft PR, mechanism shipped behind an off-by-default flag, nothing -measured.** Default is byte-identical to today. Lane spend: **$0.00** (the -item's authority was $0; no API calls, no DTU, no infrastructure created, -nothing to tear down). - -## 1. Step 0 — the free measurement that could have cancelled this item - -The item said: read the summarizer call; if it already prepends the parent's -assembled request, §2h conflict #8 resolves in favour of [P6]'s 2.4% table -figure and **there is nothing to build**. - -It does not. `_run_summary_compaction_task` built, verbatim: - -```python -request = ChatRequest( - messages=[ - Message(role="system", content=prompt), # the ~955-char prompt - Message(role="user", content=formatted), # the span, re-rendered as plain text - ], - model=self.summarization_model, -) -``` - -No parent history. No tools. Its own `role: "system"` prompt. This is the -standalone shape, so **the branch that cancels the item does not apply** and -the fix is real work. §2d's indirect evidence was right: G3 had to split out -the summarizer's own `instructions` population to get a clean hash precisely -*because* the summarizer sends its own separate system prompt. - -**What this does NOT resolve:** the 8.3–10.9% (prose) vs 2.4% (same source's -own table) conflict about the summarizer's cost *share*. Being standalone -makes the cost avoidable; it does not say how much cost there is. That number -must come from a run's own per-call table, and the follow-up item requires it. - -## 2. What shipped - -`summary_call_mode: "standalone" | "fork"`, default `"standalone"`, consulted -only when `compaction_strategy == "summary"`. - -- **`"standalone"`** — the pre-existing two-message call, unchanged. - `"inline"` is accepted as an alias (the lane brief and the work item named - the same default differently; both mean "today's behaviour"). Unknown values - warn and fall back, matching every other enum in this module. -- **`"fork"`** — the same ask re-issued as a **pure append**: - `[...the exact messages of the last request...] + [one user message: prompt + scope]`. - Pure append is the one mutation measured as a HIT under grow-only - (P4: identical-repeat 9,789 HIT, pure-append 9,789 HIT, truncation 0, - middle-drop 0). - -Three consequences, each load-bearing rather than incidental: - -1. **The span is not re-sent.** It is already inside the prefix. Re-sending it - would cost what standalone costs today *plus* the prefix — a regression - wearing an optimization's clothes. This is the single decision that - determines whether the feature is worth anything. -2. **The prompt moves into the appended `role: "user"` message.** A fork must - not add a `role: "system"` message: providers hoist every system-role - message into one top-level system block, so a per-summarization one would - rewrite the cached system prefix — the exact failure already measured for - the summary tier (cache_read 46,307 → 21,523) and the compaction notice. -3. **Scope must be stated.** Standalone scopes by construction (it can only - see the span). A fork can see everything, so the appended message names the - span: its message count plus a bounded (300-char) verbatim excerpt of its - final message as the "summarize up to here" marker. **This is a real - behavioural difference between the two modes**, not formatting — flagged as - a residual risk in the follow-up item, not hand-waved. - -New optional public seam: **`note_request_sent(messages=None, *, tools=None, -model=None)`** — see §3, which is the finding this lane most wants read. - -Observability: `last_summary_call_stats` property (`mode_requested`, -`mode_used`, `reason`, `prefix_messages`, `fork_fallbacks`) and a `call_mode` -field on `context:pre_summarize` / `context:post_summarize`, so an eval arm -can count *real* forks without patching the module. - -Size: **170 executable lines** across the new helpers (item estimated 50–70). -The overrun is entirely the refusal ladder (§3) and the staleness guard (§4); -the happy path is ~30 lines. - -## 3. The finding: this cannot be done from inside the context module alone - -**Tool specs are part of the cached prefix and this module is never handed -them.** Anthropic serializes `tools` *ahead of* the system block; OpenAI's -implicit cache matches forward from a cached entry. A summarizer request with -`tools=None` diverges from the parent at byte zero, so it hits nothing — and -because it now carries the whole conversation, it costs *more than the -standalone call it replaced*. That is not a missed win, it is a regression. - -I looked for a way to obtain them without changing any caller and found none -that is contract-backed: - -- `amplifier_core.interfaces` defines no tool-registry protocol; the - coordinator exposes `mount_points`, not tool specs in send order. -- The `llm:request` event carries `model`/`message_count`/thinking flags, and - the actual tool payload only under provider `raw: true` — where it is wire - dicts passed through `redact_secrets()`, i.e. not reconstructable into - `ToolSpec` byte-faithfully. -- The default orchestrator lives in the compiled Rust engine, so there is no - seam to read it from. - -So fork mode requires the caller to say what it sent. **Decision taken without -waiting** (per SCOPE-OUTS): add one optional public method rather than a -config knob that lets an operator *assert* alignment they cannot verify. -Passing `tools` **at all** — even `None`/`[]` for a genuinely tool-free -session — is what arms the fork; "not supplied" and "supplied as empty" are -deliberately distinguishable, because guessing between them is exactly how a -fork silently misaligns. - -**Every misalignment refuses and falls back to standalone, loudly** — WARNING -naming the precondition (once per distinct reason, so a session that can never -fork costs a handful of log lines, not one per summarization), a session -counter, and the mode actually used reported on both hooks and the stats -property. The refusals: seam never called · `summarization_model` set (a -summarizer routed elsewhere reads none of the main line's cache — deepseek -states this explicitly) · no request recorded yet · prefix ends on an assistant -turn with unanswered `tool_calls` (appending there would interleave between -`tool_use` and `tool_result`) · span absent from the recorded prefix · the -forked request fails to build. - -Falling back to *standalone* rather than skipping is deliberate: standalone is -correct and costs what it always did. Losing a summary over a cache -optimization would be strictly worse than not optimizing. - -**Consequence for the eval, stated plainly:** the S5-CRAC harness must wire -`note_request_sent()` or the treatment arm silently degrades into the control -arm and every gate is vacuous. That is the first thing the follow-up item -(`model_performance-6da`) tells the runner to verify — on a 1-run smoke, -before any spend. - -## 4. Provider asymmetry, and the staleness trap - -**Anthropic vs OpenAI need different fork sources, and the seam covers both.** -Anthropic places its cache breakpoint on the last *stable* message, walking -back past ephemeral/injected content — which is exactly where this module's -own returned view ends. So `tools` alone is enough there. OpenAI's implicit -cache measured **MISS on strict truncation** (P4), i.e. it needs a strict -superset of a cached request; an orchestrator-injected tail this module never -sees would break that. So on OpenAI the caller must also pass `messages`. -`_capture_fork_prefix` prefers the caller's record and falls back to the -module's own view. - -**The trap I built a guard for.** A caller that wires `note_request_sent()` -once (a startup helper, first turn only) would have turn 40's fork append to -turn 1's request — a guaranteed miss *and* a wasted cache write, wearing a -correct-looking API call. A `_view_serial`/`_sent_serial` pair means a -`messages` record is only used while it still describes the most recent -request served; a stale one is ignored (module view used instead), never -trusted. Tested. - -## 5. Evidence — 282 tests green (250 before, 32 new) - -`tests/test_summary_call_mode_fork.py`, written against the ways this goes -silently wrong rather than the way it is supposed to work: - -| Group | What it pins | -|---|---| -| **A — the default must not move** | The standalone request is asserted against **independently rebuilt** expected content (2 messages, system prompt then formatted span, `model` from `summarization_model`), not against itself. Plus: the fork bookkeeping is never even *written* unless armed (`_last_request_view is None` after 5 requests in both default configs) — an always-on capture would be a silent per-request list allocation on the hot path. Plus: `note_request_sent()` is fully inert when fork is not configured. | -| **B — pure append** | `sha256(fork[:-1]) == sha256(parent_view)` (G-FORK-PREFIX in unit form); exactly one appended message; **no new system message**; tools and model pinned from the seam; the span is **not** re-sent (`formatted not in appended`, and the appended message is shorter than the formatted span); the caller's `messages` record wins verbatim when supplied, including an injected tail. | -| **C — the main line is not the summarizer's scratchpad** | No `_seq` consumed; history byte-identical before/after; `_removed_seqs` unmoved; **`_last_sent_estimate` unmoved** (building the fork through `_finalize_view` would silently rewrite the hybrid meter's conservatism comparand with the summarizer's own request — this is why `_build_fork_request` calls `_strip_internal_metadata` directly); and a forked session serves views **bit-for-bit identical** to an unforked control, same `_removed_seqs`. | -| **D — refusals** | All six refusal paths fall back to the 2-message standalone request with the reason recorded; warning fires; warn-once-per-reason while the counter still counts every one; a failed fork build still produces the summary; hooks report the mode actually used; the stale-record guard prefers the fresh view. | -| **E — tool pairs** | `_select_summary_absorb_seqs` returns **identical** spans in both modes on a tool-pair-heavy history; end-to-end, no tool result is ever served without its call. The call mode changes how the summarizer is *called*, never what is *selected*. | -| **F — reset/lifecycle** | `clear()` and `set_messages()` drop all fork alignment state; the fork prefix is snapshotted at **trigger** time, not at task-scheduling time (proved by mutating `_last_request_view` while the task is in flight); the single-summarization-in-flight guard still holds under `asyncio.gather` of three concurrent requests. | - -`ruff check` clean. `ruff format` was **not** run: the repo is not -format-clean today (7 pre-existing files would be reformatted), so running it -would bury this change in unrelated noise. - -## 6. What is NOT claimed - -- **No cache win. No cost win. Nothing was measured.** G-FORK-PREFIX / - G-FORK-CACHED / G-FORK-COST / G-FORK-NOBOUNDARY are the follow-up item - (`model_performance-6da`, filed `discovered-from` this one, with the arm - design, the blocking prerequisite, and the residual risks). $0 spent here by - mandate. -- **Honest ceiling, unchanged from the item's own statement:** this reduces - the separate summarizer charge only. It does **not** touch the boundary - rebuild, which §2d measured as the dominant cost (+84% boundaries → +83% run - cost). If the summarizer's real share is the table's 2.4% rather than the - prose's 8.3–10.9%, a *perfect* fork is worth ~2% of run cost and the honest - recommendation may be "mechanism proven, not worth enabling". The follow-up - item is required to report that share from the run's own table and to say so - if it lands there. -- **Retention parity is not established.** Fork mode scopes by instruction - where standalone scopes by construction. The follow-up item reports S5 score - and `b_constraints`/`c_post_compaction` per arm so a cost win bought with a - retention loss cannot be reported as a win. -- **`reasoning_effort` / thinking configuration is not reproduced** by the - fork — it is not observable from inside a context module. If a provider keys - its cache on it, the fork misses despite byte-aligned messages. - G-FORK-CACHED is the detector; this is disclosed, not designed around. -- **The compaction-buffer reserve (condition (a) in the item) is not - implemented.** The appended instruction is ~1.2k chars, not the ~8,000-token - reserve the item contemplated, and the fork never grows the *main* line's - request. If the follow-up eval shows the fork's own request crowding the - window, that reserve is the fix, and G-FORK-NOBOUNDARY is the gate that - would catch it. - -## 7. Deliverable ledger - -| Deliverable | Status | -|---|---| -| DRAFT PR on origin, branch `lane/7k2-summary-call-fork`, default byte-identical, tests green | **DONE** | -| Prefix-stability test proving the fork does not touch the main line | **DONE** — Group C (no `_seq`, history byte-identical, `_last_sent_estimate` unmoved, forked-vs-control views identical) | -| Follow-up eval item filed via `work_file` with its arm design | **DONE** — `model_performance-6da` | -| DONE-NOTE.md in the PR body | **DONE** — this section | - -No PII, no team-internal data, no individual attribution. No merges to main. -No files touched outside this module. No infrastructure created; nothing to -tear down. diff --git a/README.md b/README.md index 632c947..8af2339 100644 --- a/README.md +++ b/README.md @@ -364,7 +364,6 @@ for the full write-up. Neither defect is inherited here: module = "context-simple" config = { compaction_strategy = "summary", # default: "progressive" - summary_call_mode = "standalone", # default; "fork" appends onto the live prefix summary_trigger = 0.60, # usage fraction that starts the async summarizer summarization_model = "...", # optional; None uses the provider default summarization_prompt_path = "...", # optional file override for the 5-section prompt @@ -440,127 +439,6 @@ absorptions cannot cycle; **hysteresis** on `summary_trigger` (arm at should be validated against the same cost metrics before this flag is enabled anywhere by default. -## Cache-safe fork of the summarization call (`summary_call_mode`) - -**Default is `"standalone"` — byte-identical to the summarizer call this -module has always made.** `"inline"` is accepted as an alias for it. Only -consulted when `compaction_strategy == "summary"`. - -### The defect - -The summarizer's request is, today, a **standalone** two-message call: its -own ~955-char `role: "system"` prompt, plus a freshly formatted plain-text -rendering of the span being absorbed. It shares **not one byte of prefix** -with the main conversation. So every token of that span is billed as -*fresh input* — while the provider is already holding that exact span warm -in the main line's cache. - -This resolves the internal conflict recorded in `00-what-we-know.md` §2d -(prose said the summarizer was "8.3–10.9% of run cost", the same source's -own table said 2.4%): whatever the share, the call is genuinely standalone, -so the cost is genuinely avoidable. It does **not** touch the boundary -rebuild, which §2d shows is the dominant cost — see "honest ceiling" below. - -### The shape - -`summary_call_mode: "fork"` re-issues the same ask as a **pure append**: - -``` -[ ...the exact messages of the last request... ] + [ one user message: prompt + scope ] -``` - -Pure append is the *one* mutation measured as a cache HIT under the -grow-only rule (probe P4: identical-repeat 9,789 HIT, **pure-append 9,789 -HIT**, strict truncation 0 MISS, middle-drop 0 MISS). - -Three things follow, and each is load-bearing: - -1. **The span is not re-sent.** It is already inside the prefix. Re-sending - it would cost exactly what standalone costs today *plus* the prefix — a - regression wearing an optimization's clothes. -2. **The prompt moves into the appended `role: "user"` message.** A - per-summarization `role: "system"` message is hoisted into the - provider's single top-level system block and rewrites the cached system - prefix — the same failure already measured for the summary tier and the - compaction notice. -3. **Scope has to be stated.** Standalone scopes by construction (it can - only see the span). A fork can see everything, so the appended message - names the span explicitly: message count, plus a bounded verbatim - excerpt of the span's final message as the "summarize up to here" - marker. This is a real behavioral difference between the two modes, not - a formatting detail. - -### The precondition you must wire: `note_request_sent()` - -Tool specs are serialized **ahead of** the system block, and this module is -handed messages, never tools. It also never sees a tail an orchestrator -injects *after* `get_messages_for_request()` returns. A fork missing either -is not an append onto the cached prefix at all. - -So fork mode requires the caller to say what it actually sent: - -```python -messages = await context.get_messages_for_request(provider=provider) -request = ChatRequest(messages=messages, tools=tools, model=model) -context.note_request_sent(messages, tools=tools, model=model) # <- every request -response = await provider.complete(request) -``` - -- Passing `tools` **at all** — even `None`/`[]` for a genuinely tool-free - session — is what arms the fork. "Not supplied" and "supplied as empty" - are deliberately distinguishable; guessing between them is how a fork - silently misaligns. -- Passing `messages` gives byte-parity with the wire, which an **implicit, - match-forward-only cache (OpenAI)** requires — its measured behavior - misses on anything that is not a strict superset of a cached request - (that is the same finding as "strict truncation → 0"). Omit it and the - fork appends to this module's own last returned view, which is what an - **explicit-breakpoint cache (Anthropic)** needs, since the breakpoint is - placed on the last *stable* message — exactly where this module's view - ends, before any ephemeral injection. -- Call it **every request**. A record that no longer describes the latest - request is ignored, not trusted: a one-time wiring would otherwise have - turn 40's fork append to turn 1's request. - -### A silently unforked fork is the failure mode that matters - -A fork that does not reproduce the parent's prefix wins nothing **and pays -for the whole conversation** — strictly worse than the standalone call it -replaces. Every misalignment therefore refuses, falls back to standalone, -and says so: a `WARNING` naming the precondition (once per distinct -reason), a session counter, and the mode actually used reported on -`context:pre_summarize` / `context:post_summarize` and via -`last_summary_call_stats`. - -The refusals: `note_request_sent()` never called · `summarization_model` -set (a summarizer routed elsewhere reads none of the main line's cache) · -no request recorded yet (first request of a session) · the prefix ends on -an assistant turn with unanswered `tool_calls` (appending there would -interleave between `tool_use` and `tool_result`) · the span is absent from -the recorded prefix · the forked request fails to build. - -### What is proven here, and what is not - -**Proven, structurally, by `tests/test_summary_call_mode_fork.py`:** the -default request is byte-identical and the fork bookkeeping is never even -written unless armed; the forked request is the parent prefix plus exactly -one `role: "user"` message (`sha256(fork[:-1]) == sha256(parent)`); no -`_seq` is consumed, history is untouched, `_last_sent_estimate` (the hybrid -meter's comparand) does not move, and a forked session serves views -bit-for-bit identical to an unforked control; tool-pair snapping is -unchanged; every refusal falls back loudly. - -**Not proven — no cache or cost win is claimed.** G-FORK-PREFIX / -G-FORK-CACHED / G-FORK-COST / G-FORK-NOBOUNDARY are a separately funded -evaluation; none of it was run here. - -**Honest ceiling, stated in advance:** this reduces the separate summarizer -charge only. It does **not** reduce the boundary rebuild, which §2d -measured as the dominant cost (+84% boundaries drives the +83% run cost). -If the summarizer really is 2.4% of run cost rather than 8.3–10.9%, then -even a perfect fork is worth ~2% — the arm design must therefore report the -summarizer's cost share from the run's own per-call table, not from prose. - ## Tool-result budget, shape, and spill **Every flag in this section defaults to a no-op.** With no configuration the diff --git a/amplifier_module_context_simple/__init__.py b/amplifier_module_context_simple/__init__.py index 9fed112..fdc71ce 100644 --- a/amplifier_module_context_simple/__init__.py +++ b/amplifier_module_context_simple/__init__.py @@ -155,66 +155,6 @@ / trigger hysteresis). OPT-IN, EXPERIMENTAL -- do not enable by default. -Cache-safe forking of the summarization call (opt-in, default off -- see -config `summary_call_mode`; only consulted when compaction_strategy == -"summary"): - • THE PROBLEM. The summarizer's request is, today, a STANDALONE two - message call: its own ~955-char system prompt plus a freshly - formatted plain-text rendering of the span being absorbed. It shares - not one byte of prefix with the main line, so every token of the span - is billed as FRESH input -- even though the provider is already - holding that exact span in a warm cache for the main conversation. - • THE SHAPE THAT FIXES IT. `summary_call_mode: "fork"` re-issues the - same ask as a PURE APPEND onto the prefix the main line already sent: - [ ...the exact messages of the last request... ] + [ one user message - carrying the summarization instruction ]. Pure append is the one - mutation measured as a cache HIT under the grow-only rule (probe P4: - identical-repeat 9,789 HIT, pure-append 9,789 HIT, strict truncation - 0 MISS, middle-drop 0 MISS). The span itself is NOT re-sent -- it is - already in the appended-to prefix; re-sending it would cost exactly - what standalone costs today PLUS the prefix, i.e. a regression. - • THE PROMPT MOVES INTO THE USER MESSAGE. The standalone call puts the - summarization prompt in a `role: "system"` message. A fork MUST NOT: - providers hoist every system-role message into a single top-level - system block, so a per-summarization system message would rewrite the - cached system prefix -- the exact failure mode already documented for - the summary tier and the compaction notice. In fork mode the prompt - rides the appended `role: "user"` message instead. - • THE MAIN LINE IS NEVER TOUCHED. The forked request is built from a - read-only snapshot. No `_seq` is consumed, nothing is appended to - `self.messages`, no sticky/removal state moves, and `_last_sent_estimate` - (the hybrid meter's conservatism comparand) is NOT rewritten -- which - is why the fork calls `_strip_internal_metadata` directly rather than - `_finalize_view`. The summary that eventually lands is produced by the - unchanged `_swap_in_pending_summary` path. - • A SILENTLY UNFORKED FORK IS THE FAILURE MODE THAT MATTERS. If the - forked request does not reproduce the parent's prefix byte-for-byte - it wins nothing AND pays for the whole conversation -- strictly worse - than standalone. So the fork is attempted ONLY when every alignment - precondition holds, and any miss falls back to the standalone call - LOUDLY (a warning naming the precondition, a counter, and the mode - actually used reported on `context:post_summarize` and in - `last_summary_call_stats`). It never silently half-forks. - • THE PRECONDITION YOU MUST WIRE. Tool specs are part of the cached - prefix (they are serialized ahead of the system block), and this - module is handed messages, never tools. So a caller that wants fork - mode MUST hand over the request it actually sent, via the optional - public `note_request_sent(messages=..., tools=..., model=...)` seam. - Without it the fork refuses (warning, once) and standalone is used. - Supplying `messages` too gives exact parity with what went on the - wire -- including any hook-injected tail the orchestrator appended - after this module returned its view, which is what an implicit, - match-forward-only cache (OpenAI) requires. With only `tools`, the - fork appends to this module's own last returned view, which is what - an explicit-breakpoint cache (Anthropic) needs, since the provider - places its breakpoint at the last STABLE message -- i.e. exactly - where this module's view ends. - • UNMEASURED. The gates (G-FORK-PREFIX / G-FORK-CACHED / G-FORK-COST / - G-FORK-NOBOUNDARY) are a separately funded eval; nothing in this - module claims a measured cache or cost win yet. What IS proven here - is structural: default byte-identity, main-line non-mutation, and - pure-append shape. - Tool-result budget and spill (opt-in, default off -- see config `tool_result_budget_tokens` / `tool_result_shape` / `tool_result_budget_by_tool` / `tool_result_exempt_tools` / @@ -348,29 +288,6 @@ COMPACTION_STRATEGY_SUMMARY, ) -# summary_call_mode config values -- HOW the summarizer's own LLM call is -# shaped when compaction_strategy == "summary". "standalone" (default) is -# byte-identical to the pre-existing behavior: its own tiny two-message -# request (system prompt + formatted span), sharing nothing with the main -# line. "fork" issues the SAME summarization ask as a PURE APPEND onto the -# prefix the main line already sent, so the provider's prompt cache can be -# read instead of paying fresh input tokens for the span. See module -# docstring "Cache-safe forking of the summarization call". -# -# "inline" is accepted as an alias for "standalone": the lane brief that -# commissioned this work named the default mode "inline" while the work -# item named it "standalone". Both mean "today's behavior, unchanged". -SUMMARY_CALL_MODE_STANDALONE = "standalone" -SUMMARY_CALL_MODE_FORK = "fork" -_VALID_SUMMARY_CALL_MODES = (SUMMARY_CALL_MODE_STANDALONE, SUMMARY_CALL_MODE_FORK) -_SUMMARY_CALL_MODE_ALIASES = {"inline": SUMMARY_CALL_MODE_STANDALONE} - -# How much of the span's final message is quoted back in the fork -# instruction as the "summarize up to HERE" boundary marker. Long enough to -# be unambiguous in a real transcript, short enough that it is never a -# material share of the appended (uncached) tail. -_FORK_BOUNDARY_EXCERPT_CHARS = 300 - # tool_result_shape config values. "head" (default) keeps the leading slice # only -- the pre-existing behavior. "head_tail" splits the budget in half # and keeps both ends with an explicit omission marker between them. See @@ -502,28 +419,6 @@ def _is_real_user_message(entry: dict[str, Any]) -> bool: """ -def _normalize_summary_call_mode(value: Any) -> str: - """Canonicalize a `summary_call_mode` config value. - - Accepts the two real modes plus the documented "inline" alias for - "standalone" (the commissioning lane brief and the work item named the - same default differently -- see _SUMMARY_CALL_MODE_ALIASES). An - unusable value logs a warning and falls back to "standalone", matching - how every other enum in this module is validated: a context manager - that refuses to start is worse than one that runs at the old default - and says so. - """ - mode = _SUMMARY_CALL_MODE_ALIASES.get(value, value) - if mode not in _VALID_SUMMARY_CALL_MODES: - logger.warning( - f"context-simple: unknown summary_call_mode {value!r} (expected " - f"one of {_VALID_SUMMARY_CALL_MODES!r}); falling back to " - f"{SUMMARY_CALL_MODE_STANDALONE!r}" - ) - return SUMMARY_CALL_MODE_STANDALONE - return mode - - async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = None): """ Mount the simple context manager. @@ -581,21 +476,8 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = boundaries / +83% run cost in the T0/T1 eval. Raising this is the cheapest lever; see README "Known issue: boundary refire". - - summary_call_mode: "standalone" (default, byte-identical to the - pre-existing summarizer call) or "fork" -- issue the - summarization ask as a pure append onto the prefix the main - line already sent, so it can be cache-read instead of paying - fresh input tokens for the span. "inline" is accepted as an - alias for "standalone". Only consulted when - compaction_strategy == "summary". Fork mode additionally - requires the caller to have called note_request_sent(); see - module docstring "Cache-safe forking of the summarization - call". - summarization_model: Model identifier passed to the summarizer's - ChatRequest (default: None, i.e. provider default). Setting it - CONFLICTS with summary_call_mode "fork" (a summarizer routed to - a different model cannot read the main line's cache) and makes - the fork fall back to standalone, loudly. + ChatRequest (default: None, i.e. provider default). - summarization_prompt_path: Path to a file overriding DEFAULT_SUMMARIZATION_PROMPT (default: None). - summarization_timeout_s: Seconds to wait for the summarizer's @@ -656,10 +538,6 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = ) compaction_strategy = COMPACTION_STRATEGY_PROGRESSIVE - summary_call_mode = _normalize_summary_call_mode( - config.get("summary_call_mode", SUMMARY_CALL_MODE_STANDALONE) - ) - context = SimpleContextManager( max_tokens=config.get("max_tokens", 200_000), compact_threshold=config.get("compact_threshold", 0.92), @@ -683,7 +561,6 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = ), compaction_strategy=compaction_strategy, summary_trigger=config.get("summary_trigger", 0.60), - summary_call_mode=summary_call_mode, summarization_model=config.get("summarization_model"), summarization_prompt_path=config.get("summarization_prompt_path"), summarization_timeout_s=config.get("summarization_timeout_s", 30.0), @@ -775,7 +652,6 @@ def __init__( compact_max_consecutive_skips: int = DEFAULT_MAX_CONSECUTIVE_SKIPS, compaction_strategy: str = COMPACTION_STRATEGY_PROGRESSIVE, summary_trigger: float = 0.60, - summary_call_mode: str = SUMMARY_CALL_MODE_STANDALONE, summarization_model: str | None = None, summarization_prompt_path: str | None = None, summarization_timeout_s: float = 30.0, @@ -841,21 +717,8 @@ def __init__( compaction boundaries (+84%) and run cost (+83%) -- see module docstring and README "Known issue: boundary refire". - summary_call_mode: "standalone" (default; the pre-existing - two-message summarizer call, byte-identical) or "fork" (the - same ask appended onto the main line's already-sent prefix - so the provider can cache-read it). "inline" is an accepted - alias for "standalone". Only consulted when - compaction_strategy == "summary". See module docstring - "Cache-safe forking of the summarization call" -- in - particular, fork mode needs note_request_sent() to have been - called, and falls back to standalone (loudly) when it has - not. summarization_model: Model identifier for the summarizer's own ChatRequest. None uses the provider's default model. - Incompatible with summary_call_mode "fork" -- a summarizer - pointed at another model reads no cache the main line wrote, - so the fork falls back to standalone rather than pretending. summarization_prompt_path: Path to a file overriding DEFAULT_SUMMARIZATION_PROMPT. None uses the built-in prompt. summarization_timeout_s: Seconds to wait for the summarizer's @@ -915,7 +778,6 @@ def __init__( ) compaction_strategy = COMPACTION_STRATEGY_PROGRESSIVE self.compaction_strategy = compaction_strategy - self.summary_call_mode = _normalize_summary_call_mode(summary_call_mode) self.summary_trigger = summary_trigger self.summarization_model = summarization_model self.summarization_prompt_path = summarization_prompt_path @@ -990,43 +852,6 @@ def __init__( self._summarization_failures: int = 0 self._summarization_task: "asyncio.Task[None] | None" = None self._summary_absorbed_count: int = 0 - # --- Cache-safe summarizer fork state (summary_call_mode == "fork") --- - # ALL of these stay None/0/False unless BOTH compaction_strategy == - # "summary" AND summary_call_mode == "fork"; nothing below is even - # written in any other configuration (see _finalize_view), which is - # what makes the default path byte-identical by construction rather - # than by inspection. - # - # `_last_request_view` is this module's own last returned view, kept - # PRE-strip (it still carries `_seq`) so the fork can verify the span - # it was asked to summarize is actually present in the prefix it is - # about to append to. `_sent_*` is the richer, optional truth handed - # over by the caller through note_request_sent(). - # - # `_view_serial` / `_sent_serial` exist for one reason: a caller - # that wires note_request_sent() ONCE (at startup, in a helper that - # only runs on the first turn) would otherwise hand the fork a - # request from turn 1 to append to on turn 40. That prefix is not - # the parent's prefix any more, so appending to it is a guaranteed - # miss AND a wasted cache write -- a silent misalignment wearing a - # correct-looking API call. Matching serials is how "the caller told - # us about THIS request" is distinguished from "the caller told us - # about SOME request, once". - self._last_request_view: list[dict[str, Any]] | None = None - self._view_serial: int = 0 - self._sent_messages: list[dict[str, Any]] | None = None - self._sent_serial: int | None = None - self._sent_tools: Any = None - self._sent_tools_supplied: bool = False - self._sent_model: str | None = None - # Observability: what the LAST summarizer call actually did, and how - # many times a requested fork had to fall back. An eval arm reads - # these to tell a real fork from a silently unforked one. - self._last_summary_call: dict[str, Any] | None = None - self._summary_fork_fallbacks: int = 0 - # Precondition names already warned about, so a session that can - # never fork logs once per reason instead of once per summarization. - self._fork_warned: set[str] = set() # Real-usage token meter state (see _on_llm_response / # _measure_working_tokens). `_last_measured_prompt_tokens` holds the # most recent real usage observed via `llm:response` @@ -1154,78 +979,6 @@ async def set_system_prompt_factory( self._system_prompt_factory = factory logger.info("System prompt factory registered - will refresh on each request") - def note_request_sent( - self, - messages: list[dict[str, Any]] | None = None, - *, - tools: Any = None, - model: str | None = None, - ) -> None: - """OPTIONAL. Tell this module what request the caller actually sent. - - This exists for exactly one reason: `summary_call_mode: "fork"` - re-issues the summarization ask as a PURE APPEND onto the prefix the - main line already sent, and this module cannot see that prefix in - full. It is handed messages; it is never handed the tool specs, and - it never sees the hook-injected tail an orchestrator may append - AFTER `get_messages_for_request()` returns. Both are part of what a - provider caches -- tool specs are serialized ahead of the system - block -- so a fork built without them is not an append onto the - cached prefix at all, and would pay full price for the whole - conversation. That is strictly worse than the standalone call it - replaces, so fork mode refuses to run without this. - - Completely inert unless BOTH compaction_strategy == "summary" AND - summary_call_mode == "fork". Never mutates history: nothing here - enters `self.messages`, consumes a `_seq`, or moves any compaction - state. Callers that do not know about it lose nothing. - - Call it EVERY request, not once. A `messages` record is only used - while it still describes the most recent request this module served - -- a one-time wiring would otherwise have turn 40's fork append to - turn 1's request, which is a guaranteed cache miss dressed up as a - correct API call. A stale record is ignored (this module's own last - view is used instead), never trusted. - - Args: - messages: The exact message array sent, if known. Gives the fork - byte-parity with the wire -- required for an implicit, - match-forward-only cache (OpenAI, whose measured behavior - misses on anything that is not a strict superset of a cached - request). Omit it and the fork appends to this module's own - last returned view instead, which is what an - explicit-breakpoint cache (Anthropic) needs, since the - breakpoint lands on the last STABLE message -- exactly where - this module's view ends, before any ephemeral injection. - tools: The tool specs sent, in the order sent. Passing this at - all -- even as None or [] for a genuinely tool-free session - -- is what arms fork mode; "not supplied" and "supplied as - empty" are deliberately distinguishable, because guessing - between them is how a fork silently misaligns. - model: The resolved model, if the caller knows it. Pins the - fork to the same model the parent used; a summarizer routed - elsewhere reads no cache the main line wrote. - """ - if messages is not None: - self._sent_messages = list(messages) - self._sent_serial = self._view_serial - self._sent_tools = list(tools) if isinstance(tools, list) else tools - self._sent_tools_supplied = True - if model is not None: - self._sent_model = model - - @property - def last_summary_call_stats(self) -> dict[str, Any] | None: - """What the most recent summarizer call actually did. - - None before the first one. Otherwise a dict with `mode_requested`, - `mode_used`, `reason` (None when the requested mode was honored), - `prefix_messages`, and `fork_fallbacks` (session-cumulative). This - is how an eval arm distinguishes a real fork from a silently - unforked one WITHOUT patching the module. - """ - return dict(self._last_summary_call) if self._last_summary_call else None - async def get_messages_for_request( self, token_budget: int | None = None, @@ -1617,19 +1370,6 @@ def _reset_summary_strategy_state(self) -> None: self._summarization_failures = 0 self._summarization_task = None self._summary_absorbed_count = 0 - # Fork state is history-derived: a reset session's old prefix is not - # a prefix of anything any more, and the caller's note_request_sent() - # facts describe a request that no longer relates to this history. - # Keeping either would be exactly the stale-alignment bug fork mode - # exists to refuse. `_summary_fork_fallbacks` is a session-cumulative - # observability counter and deliberately survives. - self._last_request_view = None - self._view_serial = 0 - self._sent_messages = None - self._sent_serial = None - self._sent_tools = None - self._sent_tools_supplied = False - self._sent_model = None async def should_compact(self) -> bool: """Check if context should be compacted. @@ -1769,14 +1509,6 @@ def _finalize_view(self, messages: list[dict[str, Any]]) -> list[dict[str, Any]] """ view = self._strip_internal_metadata(messages) self._last_sent_estimate = self._estimate_tokens(view) - # Remember the PRE-strip list (it still carries `_seq`) so a forked - # summarizer call can both reproduce this exact view byte-for-byte - # AND verify the span it was asked to summarize is present in it. - # Guarded so the default path allocates nothing new and stays - # byte-identical by construction, not by inspection. - if self._fork_armed(): - self._last_request_view = list(messages) - self._view_serial += 1 return view def _cache_aggregates(self) -> dict[str, int] | None: @@ -4030,18 +3762,9 @@ async def _maybe_trigger_summary_compaction( ) return - # Snapshot the fork prefix HERE, synchronously, rather than letting - # the background task read it whenever it happens to be scheduled. - # `_last_request_view` is rewritten on every request; a task that - # read it later would append to a prefix chosen by scheduling order. - # Capturing at trigger time makes the forked request a pure function - # of this moment -- deterministic, and therefore testable. None in - # every configuration but fork. - fork_prefix = self._capture_fork_prefix() - self._is_summarizing = True self._summarization_task = asyncio.create_task( - self._run_summary_compaction_task(seqs, fork_prefix=fork_prefix) + self._run_summary_compaction_task(seqs) ) def _select_summary_absorb_seqs(self, excess_tokens: int) -> list[int] | None: @@ -4158,284 +3881,7 @@ def result_indices(assistant_msg: dict[str, Any]) -> list[int]: return 0 return 0 # defensive; unreachable given the termination argument above - # --- Cache-safe fork of the summarizer call (summary_call_mode) -------- - # - # Everything from here to _run_summary_compaction_task is inert unless - # BOTH compaction_strategy == "summary" AND summary_call_mode == "fork". - # The default path never calls any of it (see _build_summary_request's - # first branch), which is what makes "default is byte-identical" a - # structural property rather than a claim. - - def _fork_armed(self) -> bool: - """True only when a forked summarizer call is actually configured.""" - return ( - self.compaction_strategy == COMPACTION_STRATEGY_SUMMARY - and self.summary_call_mode == SUMMARY_CALL_MODE_FORK - ) - - def _capture_fork_prefix(self) -> list[dict[str, Any]] | None: - """Snapshot the message array a forked call would append to. - - Prefers the caller's own `note_request_sent(messages=...)` record - (byte-parity with the wire, including any tail the orchestrator - injected after this module returned). Falls back to this module's - last returned view, which still ends exactly where an - explicit-breakpoint provider places its cache breakpoint. - """ - if not self._fork_armed(): - return None - source = self._last_request_view - if self._sent_messages is not None: - if self._sent_serial == self._view_serial: - source = self._sent_messages - else: - logger.debug( - "context-simple: ignoring a stale note_request_sent() " - f"message record (recorded at view {self._sent_serial}, " - f"now at view {self._view_serial}); appending to this " - "module's own last returned view instead" - ) - return list(source) if source is not None else None - - @staticmethod - def _message_identity(msg: dict[str, Any]) -> tuple[str, str, str]: - """A content-level identity for a message, used only to check span - presence when the prefix came from a caller and therefore had its - internal `_seq` already stripped.""" - return ( - str(msg.get("role", "")), - str(msg.get("content", "")), - str(msg.get("tool_call_id", "")), - ) - - def _prefix_contains_span( - self, prefix: list[dict[str, Any]], span: list[dict[str, Any]] - ) -> bool: - """Is every message of the span actually present in the prefix? - - A fork does NOT re-send the span -- the whole point is that the span - is already inside the prefix being appended to. If it is not (a - compaction removed it between the last request and this trigger), - the forked call would be asking the model to summarize text it - cannot see. Checked by `_seq` when the prefix carries them, and by - content identity when it came from a caller (post-strip). - """ - prefix_seqs = {self._extract_seq(m) for m in prefix} - {None} - span_seqs = {self._extract_seq(m) for m in span} - {None} - if span_seqs and span_seqs <= prefix_seqs: - return True - present = {self._message_identity(m) for m in prefix} - return all(self._message_identity(m) in present for m in span) - - def _fork_refusal_reason( - self, - messages_to_summarize: list[dict[str, Any]], - fork_prefix: list[dict[str, Any]] | None, - ) -> str | None: - """Why this summarization CANNOT be forked, or None if it can. - - Every branch here is a case where the forked request would not be a - true append onto the parent's cached prefix. A fork that misses pays - for the entire conversation as fresh input -- strictly worse than - the standalone call it replaces -- so each of these falls back - rather than half-forking. - """ - if not self._sent_tools_supplied: - return ( - "note_request_sent() has never been called, so the tool specs " - "the parent sent are unknown; tool specs are serialized ahead " - "of the system block, so a fork without them is not an append " - "onto the cached prefix at all" - ) - if self.summarization_model: - return ( - f"summarization_model={self.summarization_model!r} points the " - "summarizer at a different model than the main line, which " - "reads none of the cache the main line wrote" - ) - if not fork_prefix: - return ( - "no request has been recorded yet, so there is no prefix to " - "append to" - ) - if fork_prefix[-1].get("tool_calls"): - return ( - "the prefix ends on an assistant turn with unanswered " - "tool_calls; appending a user message there would interleave " - "between tool_use and tool_result" - ) - if not self._prefix_contains_span(fork_prefix, messages_to_summarize): - return ( - "the span selected for absorption is not present in the " - "recorded prefix, so a forked call could not see the text it " - "was asked to summarize" - ) - return None - - def _note_fork_fallback(self, reason: str) -> None: - """Record and (once per distinct reason) announce a refused fork. - - WARNING level and counted, never silent: a fork that quietly did not - happen is the exact failure mode that makes this treatment look like - it does not work. Once per reason, not once per call, so a session - that can never fork logs a handful of lines instead of hundreds. - """ - self._summary_fork_fallbacks += 1 - if reason in self._fork_warned: - logger.debug(f"context-simple: summarizer fork refused again ({reason})") - return - self._fork_warned.add(reason) - logger.warning( - f"context-simple: summary_call_mode='fork' requested but this " - f"summarization ran STANDALONE instead -- {reason}. The standalone " - "call is correct and costs what it always did; no cache reuse was " - "attempted. This is logged once per distinct reason." - ) - - def _span_boundary_excerpt(self, msg: dict[str, Any]) -> str: - """A short verbatim excerpt of the span's final message, used as the - 'summarize up to HERE' marker in the fork instruction.""" - content = msg.get("content", "") - if isinstance(content, list): - parts: list[str] = [] - for block in content: - if isinstance(block, dict): - text = block.get("text", "") - if text: - parts.append(str(text)) - elif hasattr(block, "text"): - parts.append(str(block.text)) - content = "\n".join(parts) - text = str(content).strip() - if not text: - # A tool_calls-only assistant turn has no text of its own; name - # the tools instead of emitting an empty, useless marker. - names = [ - str(tc.get("name") or tc.get("tool") or "") - for tc in (msg.get("tool_calls") or []) - if isinstance(tc, dict) - ] - names = [n for n in names if n] - text = ( - f"[{msg.get('role', 'unknown')} turn calling: {', '.join(names)}]" - if names - else f"[{msg.get('role', 'unknown')} turn with no text content]" - ) - if len(text) > _FORK_BOUNDARY_EXCERPT_CHARS: - text = text[:_FORK_BOUNDARY_EXCERPT_CHARS] + "..." - return text - - def _format_fork_instruction( - self, messages_to_summarize: list[dict[str, Any]], prompt: str - ) -> str: - """The single user message a forked call appends. - - Carries the summarization prompt (which in standalone mode is a - `role: "system"` message -- see module docstring for why a fork must - NOT add one) plus explicit scoping, because a fork does not re-send - the span: the model reads it from the prefix it is already holding, - so the instruction has to say which part of that prefix to - summarize. This is a REAL difference from standalone, which scopes - by construction: standalone can only see the span, a fork can see - everything and is asked to attend to the span. - """ - n = len(messages_to_summarize) - excerpt = self._span_boundary_excerpt(messages_to_summarize[-1]) - return ( - f"{prompt}\n\n" - "SCOPE OF THIS SUMMARY. Summarize ONLY the OLDEST part of the " - f"conversation above: the first {n} message(s) following the " - "system prompt -- the span about to be retired from context to " - "make room. That span ENDS with the message excerpted below. Do " - "not summarize anything after it, and do not describe this " - "instruction.\n\n" - "--- final message of the span (verbatim excerpt) ---\n" - f"{excerpt}\n" - "--- end excerpt ---" - ) - - def _build_summary_request( - self, - messages_to_summarize: list[dict[str, Any]], - fork_prefix: list[dict[str, Any]] | None, - ) -> tuple[Any, str, str | None]: - """Build the summarizer's ChatRequest. - - Returns (request, mode_actually_used, fallback_reason_or_None). The - standalone branch below is verbatim the pre-existing call and is the - ONLY branch reachable unless fork mode is both configured and - satisfiable. - """ - from amplifier_core import ChatRequest, Message - - prompt = self._get_summarization_prompt() - reason: str | None = None - - if self._fork_armed(): - reason = self._fork_refusal_reason(messages_to_summarize, fork_prefix) - if reason is None: - try: - assert fork_prefix is not None # guaranteed by the check above - return ( - self._build_fork_request( - messages_to_summarize, fork_prefix, prompt - ), - SUMMARY_CALL_MODE_FORK, - None, - ) - except Exception as e: - # A prefix message this module never created (unexpected - # content shape from a caller, say) can fail Message - # validation. Falling back keeps the summary happening at - # today's cost instead of turning a cache optimization - # into a lost summary. - reason = f"the forked request could not be built ({e!r})" - self._note_fork_fallback(reason) - - formatted = self._format_messages_for_summarization(messages_to_summarize) - request = ChatRequest( - messages=[ - Message(role="system", content=prompt), - Message(role="user", content=formatted), - ], - model=self.summarization_model, - ) - return request, SUMMARY_CALL_MODE_STANDALONE, reason - - def _build_fork_request( - self, - messages_to_summarize: list[dict[str, Any]], - fork_prefix: list[dict[str, Any]], - prompt: str, - ) -> Any: - """The forked request: the parent's prefix, then ONE appended user - message. Nothing else -- no extra system message, no re-sent span, - no reordering. - - Deliberately calls `_strip_internal_metadata` and NOT - `_finalize_view`: the latter also rewrites `_last_sent_estimate`, - the hybrid meter's conservatism comparand, which describes the view - the MAIN line sent. A summarizer call must not move it. - """ - from amplifier_core import ChatRequest, Message - - prefix_view = self._strip_internal_metadata(fork_prefix) - messages = [Message(**msg) for msg in prefix_view] - messages.append( - Message( - role="user", - content=self._format_fork_instruction(messages_to_summarize, prompt), - ) - ) - return ChatRequest( - messages=messages, - tools=self._sent_tools, - model=self._sent_model, - ) - - async def _run_summary_compaction_task( - self, seqs: list[int], fork_prefix: list[dict[str, Any]] | None = None - ) -> None: + async def _run_summary_compaction_task(self, seqs: list[int]) -> None: """Background task: call the summarizer over the message span identified by `seqs` and stash the result in `_pending_summary` for the next get_messages_for_request()/_compact_ephemeral() call to @@ -4462,29 +3908,24 @@ async def _run_summary_compaction_task( if provider is None: raise RuntimeError("no cached provider available for summary compaction") - request, call_mode, fallback_reason = self._build_summary_request( - messages_to_summarize, fork_prefix + prompt = self._get_summarization_prompt() + formatted = self._format_messages_for_summarization(messages_to_summarize) + + from amplifier_core import ChatRequest, Message + + request = ChatRequest( + messages=[ + Message(role="system", content=prompt), + Message(role="user", content=formatted), + ], + model=self.summarization_model, ) - self._last_summary_call = { - "mode_requested": self.summary_call_mode, - "mode_used": call_mode, - "reason": fallback_reason, - "prefix_messages": ( - len(request.messages) - 1 - if call_mode == SUMMARY_CALL_MODE_FORK - else 0 - ), - "fork_fallbacks": self._summary_fork_fallbacks, - } if self._hooks is not None: try: await self._hooks.emit( "context:pre_summarize", - { - "message_count": len(messages_to_summarize), - "call_mode": call_mode, - }, + {"message_count": len(messages_to_summarize)}, ) except Exception as e: logger.warning(f"Could not emit context:pre_summarize: {e}") @@ -4503,10 +3944,7 @@ async def _run_summary_compaction_task( try: await self._hooks.emit( "context:post_summarize", - { - "summary_length": len(summary_text), - "call_mode": call_mode, - }, + {"summary_length": len(summary_text)}, ) except Exception as e: logger.warning(f"Could not emit context:post_summarize: {e}") diff --git a/tests/test_summary_call_mode_fork.py b/tests/test_summary_call_mode_fork.py deleted file mode 100644 index e039f93..0000000 --- a/tests/test_summary_call_mode_fork.py +++ /dev/null @@ -1,875 +0,0 @@ -"""Adversarial tests for `summary_call_mode: "fork"`. - -STEP 0 finding this file exists to lock in: before this change the -summarizer sent a STANDALONE two-message request -- its own ~955-char -`role: "system"` prompt plus a freshly formatted plain-text rendering of -the span -- sharing not one byte of prefix with the main conversation. -Every token of the span was billed as fresh input while the provider was -already holding that exact span warm for the main line. - -`summary_call_mode: "fork"` re-issues the same ask as a PURE APPEND onto -the prefix the main line already sent. Pure append is the one mutation -measured as a cache HIT under the grow-only rule (probe P4: identical -repeat 9,789 HIT, pure append 9,789 HIT, strict truncation 0 MISS, -middle-drop 0 MISS). - -The tests below are written against the ways this can silently go wrong, -not against the way it is supposed to work: - - 1. THE DEFAULT MUST NOT MOVE. A new branch that is "usually" inert is - not inert. Group A pins the standalone request byte-for-byte and - proves the fork bookkeeping is never even written unless armed. - 2. A SILENTLY UNFORKED FORK IS THE REAL FAILURE. A fork that does not - reproduce the parent's prefix wins nothing AND pays for the whole - conversation -- strictly worse than what it replaces. Group D proves - every misalignment refuses, falls back to the standalone call, and - SAYS SO (warning + counter + reported mode). - 3. THE MAIN LINE IS NOT THE SUMMARIZER'S SCRATCHPAD. Group C proves the - fork consumes no `_seq`, appends nothing to history, does not move - the hybrid meter's `_last_sent_estimate` comparand, and leaves the - next served view byte-identical to an unforked control. - 4. TOOL PAIRS STAY WHOLE. Group E proves the absorb-boundary snapping - (the donor's exact production failure) is not perturbed by the call - mode -- the fork changes how the summarizer is CALLED, never what is - selected. -""" - -import asyncio -import hashlib -import json -import logging - -import pytest -from amplifier_core import ChatResponse, Message, TextBlock -from amplifier_module_context_simple import SimpleContextManager, mount - - -class _FakeProvider: - """Minimal stand-in for a Provider -- records every request it is - handed so tests can assert on the exact shape that would go on the - wire. Deliberately has neither `get_model_info` nor `get_info`, so - `_calculate_budget` falls back to `self.max_tokens`.""" - - def __init__(self, response_text: str = "SUMMARY TEXT"): - self.response_text = response_text - self.calls: list = [] - - async def complete(self, request): - self.calls.append(request) - return ChatResponse(content=[TextBlock(type="text", text=self.response_text)]) - - -class _Coordinator: - def __init__(self): - self.hooks = None - self.mounted = {} - - async def mount(self, kind, instance): - self.mounted[kind] = instance - - -def _tools(): - """A tool spec list shaped like what an orchestrator actually sends.""" - return [ - { - "name": "bash", - "description": "run a command", - "parameters": {"type": "object", "properties": {}}, - } - ] - - -def _digest(messages) -> str: - """Canonical hash of a message array, whatever form it arrives in.""" - dumps = [] - for m in messages: - if isinstance(m, Message): - dumps.append(m.model_dump()) - else: - dumps.append(Message(**m).model_dump()) - return hashlib.sha256( - json.dumps(dumps, sort_keys=True, default=str).encode() - ).hexdigest() - - -def _strip_timestamps(messages: list[dict]) -> list[dict]: - result = [] - for msg in messages: - meta = dict(msg.get("metadata") or {}) - meta.pop("timestamp", None) - result.append({**msg, "metadata": meta}) - return result - - -async def _await_pending_task(context: SimpleContextManager) -> None: - task = context._summarization_task - assert task is not None, "expected a background summarization task in flight" - await task - - -def _summary_manager(**overrides) -> SimpleContextManager: - """A manager whose summary trigger is reachable without needing a - 200k-token fixture. `max_tokens` is corrected per test once real - estimator usage is known (same technique the existing summary tests - use).""" - kwargs = dict( - compaction_strategy="summary", - compaction_notice_enabled=False, - protected_recent=0.5, - summary_trigger=0.3, - target_usage=0.2, - compact_threshold=0.99, # keep the outer progressive gate CLOSED - max_tokens=1_000_000, - ) - kwargs.update(overrides) - return SimpleContextManager(**kwargs) - - -async def _fill(context: SimpleContextManager, turns: int = 30) -> None: - for i in range(turns): - await context.add_message( - {"role": "user", "content": f"user turn {i} " + "x" * 40} - ) - await context.add_message( - {"role": "assistant", "content": f"assistant reply {i} " + "y" * 40} - ) - - -async def _arm_below_trigger(context: SimpleContextManager) -> list[dict]: - """Serve one request with usage BELOW summary_trigger. - - This is what records the prefix a later fork appends to -- and it is - also the honest ordering: on the very first request of a session there - is no sent prefix yet, so there is nothing to fork onto. - """ - raw = context._estimate_tokens(context.messages) - context.max_tokens = int(raw / 0.1) # usage ~0.10, below trigger 0.3 - view = await context.get_messages_for_request(provider=_FakeProvider()) - assert context._is_summarizing is False, "must not trigger below summary_trigger" - return view - - -def _cross_trigger(context: SimpleContextManager) -> None: - raw = context._estimate_tokens(context.messages) - context.max_tokens = int(raw / 0.5) # usage ~0.50: above 0.3, below 0.99 - - -# --------------------------------------------------------------------------- -# Group A -- the default must not move -# --------------------------------------------------------------------------- - - -def test_default_summary_call_mode_is_standalone(): - assert SimpleContextManager().summary_call_mode == "standalone" - assert ( - SimpleContextManager(compaction_strategy="summary").summary_call_mode - == "standalone" - ) - - -def test_inline_is_an_accepted_alias_for_standalone(caplog): - """The commissioning lane brief named the default mode "inline"; the - work item named it "standalone". Both must mean today's behavior, and - neither may produce a warning.""" - with caplog.at_level(logging.WARNING): - context = SimpleContextManager(summary_call_mode="inline") - assert context.summary_call_mode == "standalone" - assert not [r for r in caplog.records if "summary_call_mode" in r.message] - - -def test_unknown_summary_call_mode_falls_back_with_warning(caplog): - with caplog.at_level(logging.WARNING): - context = SimpleContextManager(summary_call_mode="bogus") - assert context.summary_call_mode == "standalone" - assert any("unknown summary_call_mode" in r.message for r in caplog.records) - - -@pytest.mark.asyncio -async def test_mount_threads_summary_call_mode_through(): - coordinator = _Coordinator() - await mount(coordinator, {"compaction_strategy": "summary", "summary_call_mode": "fork"}) - assert coordinator.mounted["context"].summary_call_mode == "fork" - - coordinator = _Coordinator() - await mount(coordinator, {"summary_call_mode": "inline"}) - assert coordinator.mounted["context"].summary_call_mode == "standalone" - - coordinator = _Coordinator() - await mount(coordinator, {"summary_call_mode": "nonsense"}) - assert coordinator.mounted["context"].summary_call_mode == "standalone" - - -@pytest.mark.asyncio -async def test_default_mode_summarizer_request_is_byte_identical(): - """The standalone request must remain EXACTLY what it was before this - feature existed: two messages, system prompt then formatted span, - model from `summarization_model`. Asserted against independently - rebuilt expected content, not against itself.""" - context = _summary_manager(summarization_model="gpt-test") - await _fill(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - assert len(provider.calls) == 1 - request = provider.calls[0] - assert len(request.messages) == 2 - assert request.model == "gpt-test" - assert request.tools is None - - seqs = sorted(context._pending_summary["seqs"]) - span = [m for m in context.messages if context._extract_seq(m) in set(seqs)] - assert request.messages[0].role == "system" - assert request.messages[0].content == context._get_summarization_prompt() - assert request.messages[1].role == "user" - assert request.messages[1].content == context._format_messages_for_summarization(span) - assert context.last_summary_call_stats["mode_used"] == "standalone" - assert context.last_summary_call_stats["reason"] is None - - -@pytest.mark.asyncio -async def test_default_mode_never_records_a_fork_prefix(): - """The fork bookkeeping must not merely be unused in the default mode - -- it must never be WRITTEN. An always-on capture would be a silent - per-request list allocation on the hot path.""" - for kwargs in ({}, {"compaction_strategy": "summary"}): - context = SimpleContextManager(**kwargs) - for i in range(5): - await context.add_message({"role": "user", "content": f"m{i}"}) - await context.get_messages_for_request(provider=_FakeProvider()) - assert context._last_request_view is None - assert context._sent_tools_supplied is False - assert context._summary_fork_fallbacks == 0 - assert context.last_summary_call_stats is None - - -@pytest.mark.asyncio -async def test_note_request_sent_is_inert_when_fork_is_not_configured(): - """A caller that always calls the seam must not change behavior for - every session that has not opted into forking.""" - context = _summary_manager() # standalone - await _fill(context) - context.note_request_sent(tools=_tools(), model="pinned-model") - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - request = provider.calls[0] - assert len(request.messages) == 2 - assert request.tools is None - assert request.model is None - assert context._last_request_view is None - assert context._summary_fork_fallbacks == 0 - - -@pytest.mark.asyncio -async def test_note_request_sent_never_touches_history(): - context = _summary_manager(summary_call_mode="fork") - await _fill(context, turns=3) - before = json.dumps(_strip_timestamps(context.messages), default=str) - seq_before = context._next_seq - - context.note_request_sent( - [{"role": "user", "content": "an injected tail the orchestrator added"}], - tools=_tools(), - model="m", - ) - - assert json.dumps(_strip_timestamps(context.messages), default=str) == before - assert context._next_seq == seq_before - - -# --------------------------------------------------------------------------- -# Group B -- the fork is a pure append onto what the main line actually sent -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_fork_request_is_the_parent_prefix_plus_exactly_one_user_message(): - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools(), model="pinned-model") - parent_view = await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - request = provider.calls[0] - assert context.last_summary_call_stats["mode_used"] == "fork" - assert len(request.messages) == len(parent_view) + 1, ( - "a fork appends exactly one message -- no extra system message, no " - "re-sent span" - ) - # G-FORK-PREFIX, in unit form: the fork minus its trailing message is - # byte-identical to the parent request. - assert _digest(request.messages[:-1]) == _digest(parent_view) - assert request.messages[-1].role == "user" - - -@pytest.mark.asyncio -async def test_fork_adds_no_system_message(): - """A per-summarization `role: "system"` message would be hoisted into - the provider's single top-level system block and rewrite the cached - system prefix -- the exact failure already measured for the summary - tier and the compaction notice. The prompt must ride the appended user - message instead.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools()) - parent_view = await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - request = provider.calls[0] - parent_systems = sum(1 for m in parent_view if m.get("role") == "system") - fork_systems = sum(1 for m in request.messages if m.role == "system") - assert fork_systems == parent_systems - assert context._get_summarization_prompt() in request.messages[-1].content - - -@pytest.mark.asyncio -async def test_fork_pins_tools_and_model_from_note_request_sent(): - """Tool specs are serialized ahead of the system block, and a - summarizer routed to another model reads none of the cache the main - line wrote. Both must come from what the caller says it sent.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - tools = _tools() - context.note_request_sent(tools=tools, model="pinned-model") - await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - request = provider.calls[0] - assert request.model == "pinned-model" - assert request.tools is not None and len(request.tools) == len(tools) - assert request.tools[0].name == "bash" - - -@pytest.mark.asyncio -async def test_fork_does_not_resend_the_span(): - """The entire point: the span is already inside the prefix. Re-sending - it would cost exactly what standalone costs today PLUS the prefix -- - a regression dressed as an optimization.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - appended = provider.calls[0].messages[-1].content - seqs = set(context._pending_summary["seqs"]) - span = [m for m in context.messages if context._extract_seq(m) in seqs] - assert len(span) > 3, "fixture must produce a span worth not re-sending" - formatted = context._format_messages_for_summarization(span) - assert formatted not in appended - # Only the boundary marker is quoted back, and it is bounded. - assert len(appended) < len(formatted) - - -@pytest.mark.asyncio -async def test_fork_instruction_carries_the_prompt_and_explicit_scope(): - """A fork can SEE the whole conversation, unlike standalone which can - only see the span. Scoping therefore has to be stated, and the span's - end has to be identifiable.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - appended = provider.calls[0].messages[-1].content - seqs = set(context._pending_summary["seqs"]) - span = [m for m in context.messages if context._extract_seq(m) in seqs] - - assert context._get_summarization_prompt() in appended - assert f"the first {len(span)} message(s)" in appended - assert "Do not summarize anything after it" in appended - last_text = str(span[-1]["content"])[:60] - assert last_text in appended, "the span's final message must be identifiable" - - -@pytest.mark.asyncio -async def test_fork_uses_caller_supplied_messages_verbatim_when_given(): - """An orchestrator may append hook-injected content AFTER this module - returns its view. An implicit, match-forward-only cache misses on - anything that is not a strict superset of what it cached, so when the - caller tells us what actually went on the wire, THAT is what gets - appended to.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - module_view = await _arm_below_trigger(context) - wire = [*module_view, {"role": "user", "content": "injected"}] - context.note_request_sent(wire, tools=_tools()) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - request = provider.calls[0] - assert len(request.messages) == len(wire) + 1 - assert _digest(request.messages[:-1]) == _digest(wire) - assert "injected" in request.messages[-2].content - - -# --------------------------------------------------------------------------- -# Group C -- the main line is not the summarizer's scratchpad -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_fork_consumes_no_seq_and_appends_nothing_to_history(): - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - _cross_trigger(context) - - seq_before = context._next_seq - history_before = json.dumps(_strip_timestamps(context.messages), default=str) - removed_before = set(context._removed_seqs) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - assert context.last_summary_call_stats["mode_used"] == "fork" - assert context._next_seq == seq_before, "the fork must not consume a _seq" - assert ( - json.dumps(_strip_timestamps(context.messages), default=str) == history_before - ), "the fork must not append to, reorder, or edit history" - assert context._removed_seqs == removed_before - - -@pytest.mark.asyncio -async def test_fork_does_not_move_the_hybrid_meter_comparand(): - """`_last_sent_estimate` describes the view the MAIN line sent; it is - what the hybrid meter's conservatism guard compares the next - `llm:response` against. Building the fork through `_finalize_view` - would silently rewrite it with the summarizer's own request.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - estimate_after_main_request = context._last_sent_estimate - await _await_pending_task(context) - - assert context.last_summary_call_stats["mode_used"] == "fork" - assert context._last_sent_estimate == estimate_after_main_request - - -@pytest.mark.asyncio -async def test_forked_and_unforked_sessions_serve_identical_views(): - """The call mode changes how the summarizer is CALLED. What the main - line is served must be bit-for-bit the same either way.""" - forked = _summary_manager(summary_call_mode="fork") - control = _summary_manager() - for ctx in (forked, control): - await _fill(ctx) - forked.note_request_sent(tools=_tools()) - await _arm_below_trigger(forked) - await _arm_below_trigger(control) - for ctx in (forked, control): - _cross_trigger(ctx) - - for ctx in (forked, control): - await ctx.get_messages_for_request(provider=_FakeProvider("SAME SUMMARY")) - await _await_pending_task(ctx) - - assert forked.last_summary_call_stats["mode_used"] == "fork" - assert control.last_summary_call_stats["mode_used"] == "standalone" - - for ctx in (forked, control): - ctx.compact_threshold = 0.3 - forked_view = await forked.get_messages_for_request(provider=_FakeProvider()) - control_view = await control.get_messages_for_request(provider=_FakeProvider()) - - assert _strip_timestamps(forked_view) == _strip_timestamps(control_view) - assert forked._removed_seqs == control._removed_seqs - - -# --------------------------------------------------------------------------- -# Group D -- every misalignment refuses, falls back, and says so -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_fork_without_note_request_sent_falls_back_loudly(caplog): - """Tool specs are part of the cached prefix and this module is never - handed them. Guessing "probably no tools" is precisely how a fork - silently misaligns and pays for the whole conversation.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - with caplog.at_level(logging.WARNING): - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - assert len(provider.calls[0].messages) == 2, "must be the standalone request" - stats = context.last_summary_call_stats - assert stats["mode_requested"] == "fork" - assert stats["mode_used"] == "standalone" - assert "note_request_sent()" in stats["reason"] - assert context._summary_fork_fallbacks == 1 - assert any("ran STANDALONE instead" in r.message for r in caplog.records) - - -@pytest.mark.asyncio -async def test_fork_with_summarization_model_falls_back_loudly(caplog): - context = _summary_manager(summary_call_mode="fork", summarization_model="other") - await _fill(context) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - with caplog.at_level(logging.WARNING): - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - assert len(provider.calls[0].messages) == 2 - assert provider.calls[0].model == "other", "the explicit model is still honored" - assert "different model" in context.last_summary_call_stats["reason"] - assert any("ran STANDALONE instead" in r.message for r in caplog.records) - - -@pytest.mark.asyncio -async def test_fork_on_the_first_request_of_a_session_falls_back(): - """Nothing has been sent yet, so there is no prefix to append to. This - is normal, not an error -- but it must not pretend to fork.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools()) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - assert len(provider.calls[0].messages) == 2 - assert "no request has been recorded yet" in context.last_summary_call_stats["reason"] - - -@pytest.mark.asyncio -async def test_fork_refuses_when_the_prefix_ends_on_unanswered_tool_calls(): - """Appending a user message after an assistant turn whose tool results - have not arrived interleaves between tool_use and tool_result -- the - same atomicity the compaction notice already guards.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context, turns=10) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - # The recorded prefix now ends on an assistant turn awaiting results. - context._last_request_view = [ - *context._last_request_view, - { - "role": "assistant", - "content": "", - "tool_calls": [{"id": "call-1", "tool": "bash", "arguments": {}}], - }, - ] - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - assert len(provider.calls[0].messages) == 2 - assert "unanswered" in context.last_summary_call_stats["reason"] - - -@pytest.mark.asyncio -async def test_a_stale_caller_message_record_is_ignored_not_trusted(): - """A caller that wires note_request_sent() ONCE (startup helper, first - turn only) would otherwise have turn N's fork append to turn 1's - request -- a guaranteed miss AND a wasted cache write, wearing a - correct-looking API call. The fresh module view must win.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context, turns=10) - # Turn 1: the caller records what it sent. - await _arm_below_trigger(context) - context.note_request_sent( - [{"role": "user", "content": "turn one, long ago"}], tools=_tools() - ) - # Several more turns go by without the caller telling us anything. - await _fill(context, turns=10) - fresh_view = await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - request = provider.calls[0] - assert context.last_summary_call_stats["mode_used"] == "fork" - assert _digest(request.messages[:-1]) == _digest(fresh_view) - assert not any("long ago" in str(m.content) for m in request.messages) - - -@pytest.mark.asyncio -async def test_fork_refuses_when_the_span_is_absent_from_the_prefix(): - """A fork does not re-send the span. If the recorded prefix no longer - contains it, the model would be asked to summarize text it cannot - see.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - # A prefix that is real but unrelated to the span being absorbed. - context.note_request_sent( - [{"role": "user", "content": "an unrelated conversation"}], tools=_tools() - ) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - assert len(provider.calls[0].messages) == 2 - assert "not present in the recorded prefix" in ( - context.last_summary_call_stats["reason"] - ) - - -@pytest.mark.asyncio -async def test_repeated_fallbacks_warn_once_per_reason_but_count_every_time(caplog): - """A session that can never fork should cost a handful of log lines, - not one per summarization -- while the counter still tells the truth.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context, turns=6) - - with caplog.at_level(logging.WARNING): - for _ in range(4): - context._note_fork_fallback("reason one") - context._note_fork_fallback("reason two") - - assert context._summary_fork_fallbacks == 5 - warnings = [r for r in caplog.records if "ran STANDALONE instead" in r.message] - assert len(warnings) == 2 - - -@pytest.mark.asyncio -async def test_a_failed_fork_build_falls_back_instead_of_losing_the_summary(): - """A prefix message this module never created can fail request - validation. Losing the summary over a cache optimization would be a - strictly worse outcome than paying today's price.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - await _arm_below_trigger(context) - context.note_request_sent( - [{"role": "not-a-real-role", "content": "x"}], tools=_tools() - ) - # Make the span check pass so the failure comes from request building. - context._prefix_contains_span = lambda prefix, span: True - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - assert len(provider.calls[0].messages) == 2 - assert "could not be built" in context.last_summary_call_stats["reason"] - assert context._pending_summary is not None, "the summary still happened" - - -@pytest.mark.asyncio -async def test_hooks_report_the_mode_actually_used(): - """An eval arm has to be able to count real forks without patching the - module.""" - events: list[tuple[str, dict]] = [] - - class _Hooks: - async def emit(self, event, data): - events.append((event, data)) - - context = _summary_manager(summary_call_mode="fork") - context._hooks = _Hooks() - await _fill(context) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - _cross_trigger(context) - - await context.get_messages_for_request(provider=_FakeProvider()) - await _await_pending_task(context) - - modes = { - name: data.get("call_mode") - for name, data in events - if name in ("context:pre_summarize", "context:post_summarize") - } - assert modes == { - "context:pre_summarize": "fork", - "context:post_summarize": "fork", - } - - -# --------------------------------------------------------------------------- -# Group E -- tool-pair integrity is not perturbed by the call mode -# --------------------------------------------------------------------------- - - -async def _fill_with_tool_pairs(context: SimpleContextManager) -> None: - for i in range(12): - await context.add_message( - {"role": "user", "content": f"do thing {i} " + "x" * 30} - ) - await context.add_message( - { - "role": "assistant", - "content": "working", - "tool_calls": [{"id": f"call-{i}", "tool": "bash", "arguments": {}}], - } - ) - await context.add_message( - {"role": "tool", "tool_call_id": f"call-{i}", "content": "out " + "y" * 30} - ) - - -@pytest.mark.asyncio -async def test_call_mode_does_not_change_which_span_is_selected(): - forked = _summary_manager(summary_call_mode="fork") - control = _summary_manager() - for ctx in (forked, control): - await _fill_with_tool_pairs(ctx) - - assert forked._select_summary_absorb_seqs(500) == control._select_summary_absorb_seqs( - 500 - ) - - -@pytest.mark.asyncio -async def test_fork_mode_never_serves_an_orphaned_tool_result(): - """The donor's exact production failure (a dropped `function_call` - whose `function_call_output` survived) must stay impossible in fork - mode too.""" - context = _summary_manager(summary_call_mode="fork", protected_recent=0.3) - await _fill_with_tool_pairs(context) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - _cross_trigger(context) - - await context.get_messages_for_request(provider=_FakeProvider()) - await _await_pending_task(context) - context.compact_threshold = 0.3 - view = await context.get_messages_for_request(provider=_FakeProvider()) - - call_ids = { - tc.get("id") - for m in view - for tc in (m.get("tool_calls") or []) - if isinstance(tc, dict) - } - result_ids = {m.get("tool_call_id") for m in view if m.get("role") == "tool"} - assert result_ids <= call_ids, "a tool result was served without its call" - - -# --------------------------------------------------------------------------- -# Group F -- reset -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_reset_clears_fork_alignment_state(): - """After set_messages()/clear() the recorded prefix is a prefix of - nothing, and the caller's facts describe a request unrelated to this - history. Keeping either is the stale-alignment bug fork mode exists to - refuse.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context, turns=4) - context.note_request_sent(tools=_tools(), model="m") - await _arm_below_trigger(context) - assert context._last_request_view is not None - - await context.clear() - - assert context._last_request_view is None - assert context._sent_messages is None - assert context._sent_tools is None - assert context._sent_tools_supplied is False - assert context._sent_model is None - - -@pytest.mark.asyncio -async def test_fork_state_survives_nothing_across_set_messages(): - context = _summary_manager(summary_call_mode="fork") - await _fill(context, turns=4) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - - await context.set_messages([{"role": "user", "content": "resumed session"}]) - - assert context._last_request_view is None - assert context._sent_tools_supplied is False - - -@pytest.mark.asyncio -async def test_fork_snapshot_is_taken_at_trigger_time_not_task_time(): - """The background task must append to the prefix chosen when the - trigger fired, not to whatever `_last_request_view` happens to hold - when the event loop gets around to running it.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools()) - parent_view = await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - # Simulate another request landing while the summarizer is in flight. - context._last_request_view = [{"role": "user", "content": "a later, different view"}] - await _await_pending_task(context) - - assert _digest(provider.calls[0].messages[:-1]) == _digest(parent_view) - - -@pytest.mark.asyncio -async def test_concurrent_forks_are_still_serialized_by_the_in_flight_guard(): - """Nothing about forking may weaken the single-summarization-in-flight - invariant.""" - context = _summary_manager(summary_call_mode="fork") - await _fill(context) - context.note_request_sent(tools=_tools()) - await _arm_below_trigger(context) - _cross_trigger(context) - - provider = _FakeProvider() - await asyncio.gather( - context.get_messages_for_request(provider=provider), - context.get_messages_for_request(provider=provider), - context.get_messages_for_request(provider=provider), - ) - task = context._summarization_task - if task is not None: - await task - - assert len(provider.calls) == 1 - assert context._pending_summary is not None - assert context.last_summary_call_stats["mode_used"] == "fork" From 99438c5401b5e5a82814a929d66d9855a97085ce Mon Sep 17 00:00:00 2001 From: bkrabach <702425+bkrabach@users.noreply.github.com> Date: Wed, 2 Sep 2026 17:07:51 -0700 Subject: [PATCH 3/7] Revert "feat: clear_at_least -- a worth-the-rebuild predicate in front of compaction (#2o9) (#26)" This reverts commit f851d122a834b94329a034d89b68d920561dc78f. Merge policy: main carries wins only. clear_at_least was an unproven, default-off predicate. Belongs on a branch for evaluation, not on main. --- DONE-NOTE.md | 267 ------- README.md | 194 ----- amplifier_module_context_simple/__init__.py | 423 +---------- tests/test_clear_at_least_predicate.py | 770 -------------------- 4 files changed, 5 insertions(+), 1649 deletions(-) delete mode 100644 tests/test_clear_at_least_predicate.py diff --git a/DONE-NOTE.md b/DONE-NOTE.md index 412d612..4b822d7 100644 --- a/DONE-NOTE.md +++ b/DONE-NOTE.md @@ -11,7 +11,6 @@ the other. Each note keeps its own `# DONE-NOTE - ` heading verbatim. |---|---| | `model_performance-x7p` | `protected_tool_results=0` protected ALL tool results (negative-slice bug) | | `model_performance-x1r` | tool-result budget (token-denominated, head+tail, per-tool) + spill-to-disk | -| `model_performance-2o9` | `clear_at_least` — a worth-the-rebuild predicate in front of compaction (+ summary shrink guard) | --- @@ -646,269 +645,3 @@ Artifacts written (outside the module repo, under the authorized capture path No PII, no team-internal data, no individual attribution in any output. Spill paths are the only new content class in a transcript; they are caller-configured and contain no absolute path this module invents. - - ---- - -# DONE-NOTE — W3-4 / `model_performance-2o9` - -`clear_at_least` — a worth-the-rebuild predicate in front of compaction -(context-simple), plus deepseek's summary shrink guard in the same PR. - -**Spend: $0.00.** No API calls, no eval runs, no DTU, no containers, no -infrastructure created or registered (and therefore none to tear down). -Everything below is local `pytest` / `python` on this host. The lane's spend -authority was $0 and none of it was used. - ---- - -## 1. Deliverables - -| deliverable | status | -|---|---| -| DRAFT PR on origin, branch `lane/2o9-clear-at-least`, default byte-identical, full suite green | **DONE** — [microsoft/amplifier-module-context-simple#26](https://github.com/microsoft/amplifier-module-context-simple/pull/26) (draft, rebased on `3972070`) | -| Tests for block / allow / default byte-identity / prefix stability | **DONE** — 41 new tests, 249 passed + 1 skipped total | -| Follow-up eval item filed with a Given/When/Then gate | **DONE** — `model_performance-wxs`, filed `discovered-from` this item | -| DONE-NOTE.md in the PR body | **DONE** — this note | - ---- - -## 2. The one design question the item asked me to answer honestly - -The spec (`POC-SPECS/04-clear-at-least-predicate.md`) assumes a plan/apply -split: - -``` -plan = build_compaction_plan(messages, target) -freed = tokens(messages) - tokens(apply(plan, messages)) -``` - -**That split does not exist in this module.** `_compact_ephemeral` has no plan -object: it mutates an ephemeral copy level by level (levels 1–8), threading a -running `current_tokens` through every rung, and returns through -`_finalize_compaction_with_stats` from eight different call sites. There is no -point at which a projection is available *before* the work is done. - -The GOAL says: *"if the projection is not available before the decision, say so -and implement the cheapest honest alternative, documenting the choice."* So, -stated plainly: - -**Chosen: judge the ladder's actual result, not a projection of it.** The -predicate runs at `_finalize_compaction_with_stats` — the single terminal choke -point every level returns through — and *before* that method writes any durable -effect (sticky level, stats, `context:compaction` event). If the reclaim is -below the floor, the escalation is rolled back and the baseline view is -returned. - -Three things make this the honest choice rather than a workaround: - -1. **It is exact, not estimated.** The ladder runs on an ephemeral copy that - never touches `self.messages` (`_truncate_tool_wave` does - `messages[i] = self._truncate_tool_result(msg)` — a *new* dict; - `_remove_messages_with_protection` returns a *new* list). So running it is a - simulation whose only durable output is three sticky-decision sets, which are - snapshot-and-restored. The number the predicate acts on is what the - compaction *did*, not what it might do. -2. **No second estimator was introduced.** Both sides of the comparison go - through the same `_estimate_tokens` the rest of the ladder runs on — which is - exactly what the item required ("use the existing projection the ladder - already computes; do not add a second estimator"). -3. **The cost of a refusal is CPU, not tokens.** A refused boundary costs one - wasted ladder pass over an in-memory list. The thing being avoided is a full - cold prompt-cache rebuild. The asymmetry is not close. - -**`freed` is the MARGINAL reclaim**, not the distance from raw history: - -``` -freed = tokens(view this call would have returned had it decided nothing new) - - tokens(view the ladder actually produced) -``` - -This is deliberate and load-bearing. What breaks the provider's cache is *this -view differing from the last one*, not its distance from raw history. Measured -against raw history, an already-compacted session would report a large stale -"freed" on every single call and the predicate would wave through boundaries -that free nothing — the exact failure it exists to prevent. -`test_freed_is_marginal_not_measured_against_raw_history` pins this. - ---- - -## 3. Decisions taken without waiting (per SCOPE-OUTS), and why - -| # | decision | reasoning | -|---|---|---| -| 1 | **Fail-loud = `raise RuntimeError`**, not log-and-compact-anyway | The POC's own Risks §1 says the gate is *"written so the failure mode fails the run rather than degrading quietly"*, and G-CAL-NOSKIPHANG requires the run to FAIL. A silent override would let an eval pass while the predicate had stopped applying — the one outcome the gate exists to catch. Local precedent (`_provenance_overrides`) points the other way, but that escape exists to avoid a *guaranteed* provider hard-failure; here the operator has a knob to move, and the error names it. Unreachable while the predicate is disabled. | -| 2 | **The summary swap is NOT gated by the predicate** | By the time `_swap_in_pending_summary` returns it has already appended to `self.messages` and consumed a `_seq` — irreversible. Rather than build a fragile rollback for `self.messages`, that path is governed by the shrink guard (below), which checks *before* mutating. The predicate governs the progressive ladder, which is where the boundary-refire cost actually lives. | -| 3 | **Fraction form added** (`0 < v < 1` = fraction of budget) | The GOAL says "at least N tokens (or a fraction of the window)"; the POC spec said `int` only. A hardcoded token floor silently means something different on a 200k window than a 45k one. `1.0` is read as **absolute (1 token)**, not "100% of budget" — a floor of one whole budget can never be met, so the fraction reading would turn a plausible config into a guaranteed fail-loud. | -| 4 | **`compact_max_consecutive_skips <= 0` clamps UP to 1** | Read literally, 0 means "tolerate unlimited refusals" — precisely the silent hang the cap exists to prevent. | -| 5 | **A malformed value disables the predicate with a warning** | Consistent with how this module already handles `token_meter` / `compaction_strategy` / `tool_result_shape`. Never take a session down on a config typo. | -| 6 | Config names `compact_clear_at_least` / `compact_max_consecutive_skips` | Namespaced with the existing `compact_threshold`; the spec's bare `max_consecutive_skips` would have read as unrelated to compaction. | -| 7 | **No knob for the shrink guard** | There is no defensible reason to want a summary that makes the context bigger. | -| 8 | Predicate state does **not** survive `clear()` / `set_messages()` | A skip streak accumulated against one message set says nothing about a different one; inheriting it would fail loud on the first refusal after a resume. | - ---- - -## 4. An interaction bug this lane introduced and fixed inside itself - -The branch was cut from `f47c894`, but `origin/main` had advanced two commits -(`49e2799` tool-result budget, `3972070` last-user replay). **Rebased onto -`origin/main` (3972070)**; three conflicts, all additive, resolved by keeping -both sides. - -The replay feature then created a real interaction: -`replay_last_user_on_compaction` fires once per compaction *boundary*, where a -boundary is `(_sticky_level, _summary_absorbed_count)`. **A refusal leaves both -unchanged by design** — the verdict runs before `_sticky_level` is bumped — so -on the first-ever refusal that identity still differs from the initial `None` -and would have looked like a fresh boundary. A verbatim user replay would have -been appended after a compaction that did not happen: misleading to the model, -and spending the tokens the refusal exists to save. - -Fixed with an explicit call-scoped flag (`_clear_at_least_last_refused`) rather -than by changing the replay's own boundary logic, so the fix **cannot alter -main's behaviour** — the flag is always `False` while the predicate is -disabled. Pinned by `test_a_refusal_is_not_a_boundary_for_the_last_user_replay` -and its positive counterpart. - -This is worth naming as evidence for the parallel-lane process: the bug did not -exist when this lane started and would not have been found by either lane -alone. - ---- - -## 5. Evidence - -### 5.1 Test suite - -``` -249 passed, 1 skipped in 5.57s -``` - -**41 new tests** in `tests/test_clear_at_least_predicate.py`; the 208 tests -already on `origin/main` are unmodified and green. The new tests are written -adversarially against the predicate's *own* failure mode (starvation), not just -its happy path: - -- **Blocks** — an unsatisfiable floor refuses the boundary; the returned view is - the untouched history; `context:compaction` is **not** emitted; - `context:compaction-skipped` carries freed/required/level/skips. -- **Allows** — a floor of 1 token produces a view **byte-identical** to running - with the predicate off (`test_allowed_boundary_is_identical_to_the_disabled_path`). -- **Default byte-identity** — `None`, `0`, and a negative value produce - identical views *and* identical event streams; `_clear_at_least_pending` stays - `None`, proving the guarded path is never *entered*, not merely that it agreed. -- **Rollback** — all three sticky sets, `_sticky_level`, and - `_last_compaction_stats` are unchanged after a refusal; no compaction notice - appears. -- **Prefix / `_seq` stability** — two consecutive refused calls with one turn of - growth in between share a byte-identical prefix (exact comparison, no - normalisation), and `_seq` identity is untouched. -- **Tool-pair integrity** — every `tool_calls` id in a refused view is answered - by a `tool_call_id`. -- **Starvation** — no-op calls never count as skips; the cap raises with a - message naming freed/required/protected set; state is rolled back *before* the - raise; the streak resets on an accepted compaction and on clear/resume. -- **Floor resolution** — 12 parametrised cases covering None/0/negative/int/ - fraction/1.0-boundary/zero-budget/malformed. -- **Shrink guard** — larger summary refused (history untouched, span not - recorded as removed, **not** counted as a summarizer failure); *equal-sized* - summary refused (the `>=` boundary exercised directly by searching for a - text that prices exactly at the span's estimate); genuinely smaller summary - still swaps. - -### 5.2 Default byte-identity vs `origin/main` (the honest stash-compare) - -Captures: -`/home/bkrabach/dev/openai-evals-team-ci/.amplifier/evaluation/treatment-validation/20260902-2o9-clear-at-least/` -(`byte_identity_check.py`, `baseline_origin_main__init__.py`, -`treatment__init__.py`, `byte_identity_result.txt`, -`negctl_predicate_on_by_default__init__.py`, `negative_control_result.txt`). - -Both module versions are loaded **in one process** and driven through identical -scenarios with identical inputs. Non-determinism is *eliminated*, not normalised -away: message timestamps are frozen to a fixed string after history is built, so -the two runs differ in nothing but the module source. Compared per call: the -returned view, every emitted hook event (name + payload), all three sticky -decision sets, `_sticky_level`, and `_last_compaction_stats`. - -**11 of 11 scenarios byte-identical:** - -``` -IDENTICAL default IDENTICAL tool_result_head_tail -IDENTICAL token_meter_actual IDENTICAL replay_last_user -IDENTICAL token_meter_hybrid IDENTICAL notice_disabled -IDENTICAL summary_strategy_no_provider IDENTICAL protected_tool_results_zero -IDENTICAL tool_result_budget IDENTICAL larger_budget - IDENTICAL system_prompt_factory -ALL SCENARIOS BYTE-IDENTICAL -``` - -The scenario set deliberately covers **every other opt-in feature**, not just -the default path: "default byte-identical" must also mean "does not perturb a -feature that was already opt-in". - -**The check is not vacuous.** A negative control that changes exactly one line — -`DEFAULT_CLEAR_AT_LEAST = None` → `20_000` — diverges immediately on the same -harness, and does so by raising the fail-loud error, which doubles as a live -demonstration of its content: - -``` -RuntimeError: context-simple: compact_clear_at_least=20000 could not be -satisfied 3 consecutive times (cap: 3). The compaction ladder reached level 8 -and freed only 7,698 tokens against a required floor of 20,000. Protected set -holding the floor: protected_tool_results=1 (of 30 tool results in the view), -protected_recent=20% of 124 messages. Baseline view is 7,852 tokens against a -budget of 1,200. Lower compact_clear_at_least, lower -protected_recent/protected_tool_results, or raise the budget -- compacting -harder will not help. -``` - -(That fixture's budget is 1,200 tokens, so a 20k floor is unsatisfiable by -construction — i.e. exactly the misconfiguration the fail-loud path exists for, -and the message names the right knobs.) - ---- - -## 6. What is NOT claimed - -**No workload measurement exists. No performance claim rides into source, the -PR body, or the README** (§5 rule 6). - -- The mechanism is implemented and unit-tested. Its effect on boundary count, - cost, waste, or latency on a real workload is **unmeasured**. -- The eval is filed as **`model_performance-wxs`** with gates pre-registered - *before* any spend: `G-CAL-BOUNDARIES` (monotonic dose-response across - `{null, 10k, 20k, 40k}`, n≥3 — a flat response falsifies the mechanism), - `G-CAL-NOSKIPHANG`, `G-CAL-LATENCY` (the pre-registered **win**), - `G-CAL-COST` and `G-CAL-WASTE` (non-regression only), `G-CAL-SKIPCOUNT` (an - instrument check that catches a run where every floor was trivially - satisfiable), and the Anthropic guardrail from raw wire fields. -- **A cost reduction is deliberately NOT pre-registered as a win.** §2b is - explicit that fewer boundaries "buys latency, not money" (`cad-fewer`: −29% - requests, −14% wall, *same* cost, same quality), and the underlying fit - (`waste ≈ 36.3 − 0.357 × boundaries`, r = −0.586, 12 points) is suggestive, - not established. -- **A retention gate is rejected as vacuous** here as everywhere: `b_constraints` - 40/40 and `c_post_compaction` 20/20 in every run of every arm across probes - 1–6. Deferred to `model_performance-cb2`. -- **The 20,000 starting value is not pinned.** It derives from an 18,458-token - head computed through a 4.59 chars/token constant that is tokenizer- and - model-version-specific (DESIGN-SPACE.md C-4). The eval item requires - re-deriving the head *in tokens* from provider usage first. The README says so - too. -- **Do not run this on the bare estimator.** The default `token_meter: - "estimate"` is `len(str)//4`, never reconciled against provider usage and ~2× - off in production sessions; a predicate acting on a 2× off number refuses the - wrong boundaries. `model_performance-q69` landed the provider-anchored hybrid - meter with provenance precisely so this predicate has an anchored number. - README and the eval item both state `token_meter: "hybrid"` as the operating - condition. - -**Known limitation, disclosed:** a refused call pays one wasted ladder pass -(CPU over an in-memory list, no tokens). Also, like the rest of this module's -shared mutable state, the predicate's call-scoped state assumes -`get_messages_for_request` is not re-entered concurrently — a pre-existing -property of the module, not new exposure, but now with one more field. - -No PII, no team-internal data, no individual attribution in any output. No -merges to main; no files touched outside this module. diff --git a/README.md b/README.md index 8af2339..fc62917 100644 --- a/README.md +++ b/README.md @@ -721,200 +721,6 @@ of every run), so it cannot show a retention difference in either direction. Do not enable by default, and do not claim a quality benefit, until a discriminating eval has run. -## Worth-the-rebuild predicate (`compact_clear_at_least`) - -**Default `null` (disabled). Byte-identical to before this feature existed.** - -### The problem - -The compaction trigger fires on a usage *threshold* (`compact_threshold`, -default 0.92) and **never asks how many tokens the compaction will actually -free**. - -That omission is not free, because every compaction *shrinks* the request, and -a shrink is a guaranteed cold prompt-cache rebuild on the OpenAI path. The -measured mechanism (micro-probe v3, 3 reps, ~9.8k-token payloads, validity gate -passed): identical repeat **9,789 cache_read HIT**, pure append **9,789 HIT**, -and a **byte-identical strict prefix of the cached request — 0, MISS**. The -cache matches forward from a cached entry, never backward into one. So a -boundary that frees 3k tokens still pays a full rebuild of an ~18.4k-token -pinned head plus everything after it. We take that trade silently, every time. - -What the omission costs, measured: the `cad-deep` arm set `target_usage: 0.15` -— a 6,750-token target *below* the ~32k system floor — so compaction escalated -to max level on every request: **25 boundaries** (more than stock's 21.6), -prefix retention **0.0%**, **$3.16** against stock's **$2.58**, and no quality -upside. A worth-the-rebuild predicate would have refused every one of them. - -### What it does - -Lifted from Anthropic's context-editing API, whose parameter of the same name -is documented as: *"If the API can't clear at least the specified amount, the -strategy will not be applied. This helps determine if context clearing is worth -breaking your prompt cache."* Vendor-agnostic — implemented here client-side, -with no API support required. - -```yaml -compact_clear_at_least: 20000 # absolute token floor -compact_clear_at_least: 0.15 # or a fraction of the budget -compact_max_consecutive_skips: 3 # fail loud after this many refusals -``` - -| value | meaning | -|---|---| -| `null` (default), `0`, negative | disabled — today's behaviour exactly | -| int `>= 1` | absolute token floor | -| float in `(0, 1)` | that fraction of the per-call budget | -| unparseable | disabled, with a logged warning (never raises on config alone) | - -`1.0` is deliberately read as **absolute (1 token)**, not "100% of budget": a -floor of one entire budget can never be met, so reading it as a fraction would -turn a plausible-looking config into a guaranteed fail-loud. - -### What `freed` means here, precisely - -This module's ladder has **no separable plan/apply split** — it mutates an -ephemeral copy level by level, threading a running `current_tokens` through -every rung. So the predicate does not build a second estimator or a projection. -It compares two counts the ladder already produces: - -``` -freed = tokens(view this call would have returned had it decided nothing new) - - tokens(view the ladder actually produced) -``` - -That is the **marginal** reclaim of this boundary, and it is the right number: -what breaks the provider's cache is this view differing from the last one, not -its distance from raw history. (A predicate measured against raw history would -report a large stale "freed" on every call of an already-compacted session and -approve boundaries that free nothing.) Both sides go through the same -`_estimate_tokens` the rest of the ladder runs on, recomputed exactly rather -than read off the running counter, which carries small deliberate -approximations (e.g. the flat ~18-token stub charge at level 8). - -### On refusal - -The escalation is **rolled back completely**: the sticky truncate/remove/stub -decisions recorded during it are discarded, `_last_compaction_stats` is left -untouched (so the tail compaction notice stays byte-stable across a skip), -`_sticky_level` does not move, **no `context:compaction` event is emitted**, and -a `context:compaction-skipped` event carries `freed_tokens`, `required_tokens`, -`level_reached`, `consecutive_skips`, `baseline_tokens`, `budget`, and the -protected-set knobs. The returned view is the unchanged baseline, so a refusal -is strictly append-only with respect to the previous request — which is the -entire point of refusing. - -A refusal is also **not** a compaction boundary for `replay_last_user_on_compaction`: -the replay fires once per boundary identified by `(sticky_level, -summary_absorbed_count)`, and a refusal leaves both unchanged by design, so -without an explicit check the first-ever refusal would still look like a fresh -boundary and replay after a compaction that did not happen. - -### Escalation path: it fails loud, it does not hang - -This predicate can starve compaction. If it refuses and usage keeps climbing, -the next request is larger and the predicate will normally pass — but if the -plan can *never* free the floor (the protected set is too large, or the floor -is simply misconfigured), skipping forever would end in an opaque provider -context-overflow error, and quietly compacting anyway would let a run "pass" -while the predicate had silently stopped applying. - -So after `compact_max_consecutive_skips` (default 3) **consecutive** refusals it -raises `RuntimeError`, naming what the ladder freed, what was required, the -level it reached, the baseline size against the budget, and the protected set -holding the floor — i.e. which knob to actually move: - -``` -context-simple: compact_clear_at_least=20000 could not be satisfied 3 -consecutive times (cap: 3). The compaction ladder reached level 8 and freed -only 7,698 tokens against a required floor of 20,000. Protected set holding -the floor: protected_tool_results=1 (of 30 tool results in the view), -protected_recent=20% of 124 messages. Baseline view is 7,852 tokens against a -budget of 1,200. Lower compact_clear_at_least, lower -protected_recent/protected_tool_results, or raise the budget -- compacting -harder will not help. -``` - -Two deliberate details: - -- **Calls that decided nothing are never judged and never count as a skip.** - Once sticky state alone keeps the view under threshold, the ladder returns - early having refused nothing; counting those would fail loud on a session - that is behaving perfectly. -- **A cap of `0` clamps up to `1`, not down to "never fail".** Read literally, - 0 means "tolerate unlimited refusals" — exactly the silent hang the cap - exists to prevent. -- The streak does **not** survive `clear()` or `set_messages()`: a streak - accumulated against one message set says nothing about a different one. - -### Why it is not shipped on by default - -`freed` is a token count, and in the default `token_meter: "estimate"` mode it -comes from the `len(str)//4` heuristic — never reconciled against real provider -usage, and roughly 2× off in production sessions. **A predicate that refuses -boundaries based on a number that is 2× off will refuse the wrong boundaries.** -The honest resolution already shipped in this module: `token_meter: "hybrid"` -anchors on the provider's own reported total and carries provenance -(`kind ∈ {usage, estimated, none}`). **Run this predicate with -`token_meter: "hybrid"`, not on the bare estimator.** - -No workload evaluation has been run. The pre-registered gates are in the -follow-up item: boundary count must fall **monotonically** across -`{null, 10k, 20k, 40k}` at n≥3 (a flat response falsifies the mechanism), -`max_consecutive_skips` must never be reached, cost must not rise, and re-billed -waste must not rise. **A cost *reduction* is deliberately not pre-registered as -a win**: fewer boundaries measurably "buys latency, not money" -(`cad-fewer`: −29% requests, −14% wall, same cost, same quality), and the -underlying fit (`waste ≈ 36.3 − 0.357 × boundaries`, r = −0.586 over 12 points) -is suggestive, not established. A retention gate on S5-CRAC would be **vacuous** -(40/40 constraints and 20/20 post-compaction in every run of every arm across -probes 1–6) and is deferred. - -The suggested starting value of **20,000** comes from the observed pinned -`cache_read` head of 18,458 tokens — but that number was derived through a -4.59 chars/token constant that is tokenizer- and model-version-specific. -**Re-derive the head in tokens from real provider usage before pinning it.** - -## Summary shrink guard - -**Always on when `compaction_strategy: "summary"`. No config knob.** - -The rolling summarizer previously swapped in whatever the summarizer returned. -This refuses a summary that is **not smaller** than the span of messages it -replaces — including exactly equal, since paying a cache rebuild to swap content -for content of identical cost buys nothing. - -Lifted from deepseek-harness's `compaction-basic` region check: *"a summary that -is not smaller than what it replaces is a silent cost regression, and they -simply refuse it."* We had nothing like it, and a terse span plus a verbose -5-section summary is entirely capable of growing. - -On refusal the pending summary is **discarded** and the pass falls back to -progressive compaction — the same graceful path a stale summary already takes, -and explicitly **not** counted as a summarizer failure. Both sides are measured -as full message dicts through the same `_estimate_tokens`, so the comparison is -apples-to-apples. - -There is no knob because there is no defensible reason to want a summary that -makes the context bigger. - -### Byte-identity evidence - -`compact_clear_at_least: null` (the default) was verified against -`origin/main` by loading both module versions in one process and driving them -through identical scenarios with identical inputs (message timestamps frozen, so -the two runs differ in nothing but the module source), comparing every returned -view, every emitted hook event, the full sticky-decision state, and -`_last_compaction_stats` after every call. - -**11 of 11 scenarios byte-identical**: `default`, `token_meter_actual`, -`token_meter_hybrid`, `summary_strategy_no_provider`, `tool_result_budget`, -`tool_result_head_tail`, `replay_last_user`, `notice_disabled`, -`protected_tool_results_zero`, `larger_budget`, `system_prompt_factory`. - -The check is **not vacuous**: a negative control that flips only the default to -`20_000` diverges immediately and loudly on the same harness. - ## Dependencies - `amplifier-core>=1.0.0` diff --git a/amplifier_module_context_simple/__init__.py b/amplifier_module_context_simple/__init__.py index fdc71ce..919b07c 100644 --- a/amplifier_module_context_simple/__init__.py +++ b/amplifier_module_context_simple/__init__.py @@ -55,66 +55,6 @@ • Ported from amplifier-module-context-handoff's proven `_on_llm_response` meter. See README "Real-usage token meter" for the full rationale. -Worth-The-Rebuild Predicate (opt-in, default off -- see config -`compact_clear_at_least`): - • THE PROBLEM IT SOLVES: the compaction trigger fires on a usage - THRESHOLD (`compact_threshold`) and never asks how many tokens the - compaction will actually free. Every compaction shrinks the request, - and on the OpenAI path a shrink is a guaranteed full cold rebuild of - the prompt cache (the cache matches forward from a cached entry, never - backward into one -- so a strict prefix of a cached request MISSES). - A boundary that frees 3k tokens therefore pays a full rebuild of an - ~18k-token pinned head plus everything after it to buy very little. - We currently take that trade silently, every time. - • THE FIX, lifted from Anthropic's context-editing API: `clear_at_least` - -- "If the API can't clear at least the specified amount, the strategy - will not be applied. This helps determine if context clearing is worth - breaking your prompt cache." Vendor-agnostic; implemented here - client-side, no API support required. - • `compact_clear_at_least: None` (DEFAULT) or `0` disables the predicate - entirely: not evaluated, no new event, no behavioural difference. The - default path is byte-identical to before this feature existed. - • An int >= 1 is an absolute token floor. A float in (0, 1) is a - FRACTION OF THE BUDGET, resolved per call (so it tracks the model's - real window rather than a hardcoded number). - • WHAT "freed" MEANS HERE, precisely: this module's ladder has no - separable plan/apply split -- it mutates an ephemeral copy level by - level, threading a running `current_tokens` through every rung. So the - predicate does not estimate a projection; it compares the count the - ladder ALREADY computes for the view it produced against the count of - the view this call would have returned had it decided nothing new - (the sticky baseline). That delta is the MARGINAL reclaim of this - boundary -- which is the right number, because what breaks the cache - is this view differing from the last one, not the distance from raw - history. It is exact for the ladder's own units, not a second - estimator (see _clear_at_least_required / _finalize_compaction_with_stats). - • ON REFUSAL: the new escalation is rolled back (the sticky - truncate/remove/stub decisions recorded during it are discarded), the - baseline view is returned unchanged, and `context:compaction-skipped` - is emitted. No `context:compaction` event fires, because no compaction - happened. `_last_compaction_stats` is untouched, so the tail notice - stays byte-stable across a skip. - • ESCALATION PATH -- this predicate can starve compaction, so it must - not become a silent hang. After `compact_max_consecutive_skips` - (default 3) consecutive refusals it FAILS LOUD: raises RuntimeError - naming what the ladder freed, what was required, the level it reached, - and the protected-set size that is holding the floor. Degrading - quietly instead would let an eval pass while the predicate had - silently stopped applying, which is the one outcome its gate - (G-CAL-NOSKIPHANG) is written to catch. Unreachable while the - predicate is disabled. - • Calls where sticky state alone was already sufficient (no NEW decision - made) are not judged and never count as a skip -- nothing was refused. - -Summary Shrink Guard (active whenever `compaction_strategy == "summary"`): - • Refuses to swap in a rolling summary that is not SMALLER than the span - of messages it replaces. A summary that grows the context is a silent - cost regression -- it pays a cache rebuild to make the request bigger. - • Lifted from deepseek-harness's compaction-basic region check. The - pending summary is discarded (the same graceful path a stale summary - already takes) and this pass falls back to progressive compaction. - • No config knob: there is no defensible reason to want the opposite. - Summary Compaction Strategy (opt-in, default off -- see config `compaction_strategy`): • `compaction_strategy: "progressive"` (default) is this module's @@ -305,13 +245,6 @@ # tokens rather than chars. _TOOL_RESULT_CHARS_PER_TOKEN = 4 -# Worth-the-rebuild predicate defaults (see module docstring). The predicate -# is DISABLED by default -- `None` (and `0`) mean "today's behaviour exactly". -# The skip cap exists solely so a predicate that can never be satisfied fails -# loud instead of looping; it is unreachable while the predicate is disabled. -DEFAULT_CLEAR_AT_LEAST = None -DEFAULT_MAX_CONSECUTIVE_SKIPS = 3 - # The summary message's envelope source tag and metadata type marker. The # envelope is what makes foundation's is_real_user_message() classify this # role="user" message as NOT a real user turn (see module docstring); the @@ -450,17 +383,6 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = falling back to the estimator before then. An unrecognized value falls back to "estimate" with a logged warning rather than crashing mount(). See module docstring. - - compact_clear_at_least: Worth-the-rebuild predicate in front of - compaction (default: None = disabled, today's behaviour - exactly). An int >= 1 is an absolute token floor; a float in - (0, 1) is a fraction of the budget. A compaction that would - free less than this is REFUSED and rolled back rather than - paying a full prompt-cache rebuild for a small reclaim. See - module docstring "Worth-The-Rebuild Predicate". - - compact_max_consecutive_skips: How many consecutive predicate - refusals are tolerated before failing loud with a RuntimeError - (default: 3). Only consulted when compact_clear_at_least is - enabled; unreachable otherwise. - compaction_strategy: "progressive" (default) or "summary". See module docstring "Summary compaction strategy". An unrecognized value falls back to "progressive" with a logged @@ -553,12 +475,6 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = compaction_notice_min_level=config.get("compaction_notice_min_level", 1), output_reserve_fraction=config.get("output_reserve_fraction", 0.5), token_meter=token_meter, - compact_clear_at_least=config.get( - "compact_clear_at_least", DEFAULT_CLEAR_AT_LEAST - ), - compact_max_consecutive_skips=config.get( - "compact_max_consecutive_skips", DEFAULT_MAX_CONSECUTIVE_SKIPS - ), compaction_strategy=compaction_strategy, summary_trigger=config.get("summary_trigger", 0.60), summarization_model=config.get("summarization_model"), @@ -648,8 +564,6 @@ def __init__( compaction_notice_min_level: int = 1, output_reserve_fraction: float = 0.5, token_meter: str = TOKEN_METER_ESTIMATE, - compact_clear_at_least: int | float | None = DEFAULT_CLEAR_AT_LEAST, - compact_max_consecutive_skips: int = DEFAULT_MAX_CONSECUTIVE_SKIPS, compaction_strategy: str = COMPACTION_STRATEGY_PROGRESSIVE, summary_trigger: float = 0.60, summarization_model: str | None = None, @@ -690,20 +604,6 @@ def __init__( session -- see module docstring "Real-Usage Token Meter"). An unrecognized value falls back to "estimate" with a logged warning rather than raising. - compact_clear_at_least: Worth-the-rebuild predicate in front of - compaction. None (default) or 0 disables it entirely -- - byte-identical to before this feature existed. An int >= 1 - is an absolute token floor; a float in (0, 1) is a fraction - of the per-call budget. A compaction whose MARGINAL reclaim - is below the floor is refused and rolled back rather than - paying a full prompt-cache rebuild for a small reclaim. See - module docstring "Worth-The-Rebuild Predicate". - compact_max_consecutive_skips: Consecutive predicate refusals - tolerated before failing loud with RuntimeError (default 3). - Only consulted when compact_clear_at_least is enabled. A - value <= 0 is treated as 1 (refuse once, then fail) rather - than as "never fail", because "never fail" is the silent - hang this cap exists to prevent. compaction_strategy: "progressive" (default, byte-identical to pre-existing behavior) or "summary" -- see module docstring "Summary compaction strategy". An unrecognized value falls @@ -764,12 +664,6 @@ def __init__( ) token_meter = TOKEN_METER_ESTIMATE self.token_meter = token_meter - self.compact_clear_at_least = compact_clear_at_least - # A cap of 0 or less would mean "tolerate unlimited refusals", i.e. - # exactly the silent starvation this cap exists to prevent. Clamp to - # 1 (refuse once, then fail loud) rather than honouring a value that - # disables the only safety net the predicate has. - self.compact_max_consecutive_skips = max(1, int(compact_max_consecutive_skips)) if compaction_strategy not in _VALID_COMPACTION_STRATEGIES: logger.warning( f"context-simple: unknown compaction_strategy {compaction_strategy!r} " @@ -828,22 +722,6 @@ def __init__( self._last_replayed_boundary: tuple[int, int] | None = None self._hooks = hooks self._last_compaction_stats: dict[str, Any] | None = None - # --- Worth-the-rebuild predicate state (compact_clear_at_least) --- - # `_clear_at_least_skips` counts CONSECUTIVE refusals; any accepted - # compaction resets it to 0. `_clear_at_least_pending` is call-scoped - # state handed from _compact_ephemeral to its single terminal choke - # point (_finalize_compaction_with_stats): the baseline view this - # call would have returned had it decided nothing new, that view's - # token count, and the sticky-decision snapshot to roll back to. It - # is always None outside a single _compact_ephemeral call, and stays - # None entirely while the predicate is disabled. - self._clear_at_least_skips: int = 0 - self._clear_at_least_pending: dict[str, Any] | None = None - # True only for the remainder of a call whose escalation the - # predicate refused. Read by get_messages_for_request() so a - # refusal is not mistaken for a compaction boundary by the - # last-user replay. Always False while the predicate is disabled. - self._clear_at_least_last_refused: bool = False # --- Summary compaction strategy state (compaction_strategy == "summary") --- # Unused, and never touched, in the default "progressive" mode. self._cached_provider: Any = None @@ -1169,17 +1047,7 @@ async def get_messages_for_request( # suppressed (tail unsafe) or the replay lands first and the # notice's own guard then sees the replay -- a plain user # message with no tool_calls -- and proceeds correctly. - # A refused compaction is NOT a boundary. Without this check the - # replay would fire on a call where nothing was shed at all -- - # `_current_compaction_boundary()` is unchanged by a refusal (by - # design: the verdict runs before `_sticky_level` is bumped), so - # on the first-ever refusal it still differs from the initial - # `None` and would look like a fresh boundary. Appending a - # verbatim user replay after a compaction that did not happen - # would both mislead the model and spend tokens the refusal - # exists to save. Always False while the predicate is disabled, - # so this cannot change existing behaviour. - if self.replay_last_user_on_compaction and not self._clear_at_least_last_refused: + if self.replay_last_user_on_compaction: self._maybe_append_last_user_replay(compacted, effective_budget) if self.compaction_notice_enabled and self._last_compaction_stats: @@ -1324,7 +1192,6 @@ async def set_messages(self, messages: list[dict[str, Any]]) -> None: self._sticky_level = 0 self._last_compaction_stats = None self._last_replayed_boundary = None - self._reset_clear_at_least_state() self._reset_summary_strategy_state() logger.info(f"Restored {len(messages)} messages to context") @@ -1347,7 +1214,6 @@ async def clear(self) -> None: self._tool_name_by_call_id = {} self._spilled_paths = set() self._last_replayed_boundary = None - self._reset_clear_at_least_state() self._reset_summary_strategy_state() logger.info("Context cleared") @@ -1860,194 +1726,6 @@ def _apply_sticky_decisions( result.append(dict(msg)) return result - # --- Worth-the-rebuild predicate helpers (compact_clear_at_least) --- - # - # See the module docstring section "Worth-The-Rebuild Predicate" for the - # why. In short: every compaction shrinks the request, and a shrink is a - # guaranteed cold prompt-cache rebuild on the OpenAI path, so a boundary - # that frees very little is close to pure loss. This predicate refuses - # those boundaries. It is DISABLED by default and, while disabled, - # `_clear_at_least_pending` is never populated and no code path below the - # `required > 0` check is ever entered. - - def _reset_clear_at_least_state(self) -> None: - """Drop predicate state -- called from clear() and set_messages(). - - The consecutive-skip streak is per-conversation: a resumed or cleared - session has a fresh message set, so a streak accumulated against the - old one says nothing about the new one and must not be inherited (it - would otherwise fail loud on the first refusal after a resume). - """ - self._clear_at_least_skips = 0 - self._clear_at_least_pending = None - self._clear_at_least_last_refused = False - - def _clear_at_least_required(self, budget: int) -> int: - """Resolve `compact_clear_at_least` to an absolute token floor for - THIS call, or 0 meaning the predicate is disabled. - - None / 0 / negative -> 0 (disabled; today's behaviour exactly) - 0 < value < 1 -> that FRACTION of `budget`, resolved per call - value >= 1 -> that many tokens, absolutely - - The fraction form exists because a hardcoded token floor silently - means something completely different on a 200k window than on a 45k - one; a fraction tracks the real window. - - The 1.0 boundary is deliberately read as ABSOLUTE, not as "100% of - budget": a floor of one entire budget can never be met by any - compaction, so reading it as a fraction would turn a plausible-looking - config into a guaranteed fail-loud. A malformed value disables the - predicate with a warning rather than raising -- consistent with how - this module handles every other unrecognized config value. - """ - raw = self.compact_clear_at_least - if raw is None: - return 0 - try: - value = float(raw) - except (TypeError, ValueError): - logger.warning( - f"context-simple: compact_clear_at_least={raw!r} is not a number; " - f"disabling the worth-the-rebuild predicate for this session " - f"(compaction behaves exactly as if it were unset)." - ) - return 0 - if value <= 0: - return 0 - if value < 1: - return int(budget * value) if budget > 0 else 0 - return int(value) - - def _snapshot_sticky_decisions(self) -> dict[str, set[int]]: - """Copy the three sticky decision sets so a refused escalation can be - rolled back exactly. - - These sets are the ONLY durable state a progressive escalation writes - before it reaches `_finalize_compaction_with_stats` (which is where - the predicate runs). `_sticky_level`, `_last_compaction_stats` and the - `context:compaction` event are all written AFTER the predicate, so a - refusal leaves no trace of them at all -- which is what keeps the tail - compaction notice byte-stable across a skip. - """ - return { - "removed": set(self._removed_seqs), - "truncated": set(self._truncated_seqs), - "stubbed": set(self._stubbed_seqs), - } - - def _restore_sticky_decisions(self, snapshot: dict[str, set[int]]) -> None: - """Undo every sticky decision recorded since `snapshot` was taken.""" - self._removed_seqs = set(snapshot["removed"]) - self._truncated_seqs = set(snapshot["truncated"]) - self._stubbed_seqs = set(snapshot["stubbed"]) - - async def _clear_at_least_verdict( - self, - final_messages: list[dict[str, Any]], - max_level_reached: int, - ) -> list[dict[str, Any]] | None: - """Judge a completed escalation against the worth-the-rebuild floor. - - Returns None when the compaction is ACCEPTED (the caller finalizes - normally), or the baseline view to return INSTEAD when it is refused. - Raises RuntimeError when the consecutive-refusal cap is reached. - - `freed` is the MARGINAL reclaim of this boundary: the count of the - view this call would have returned had it decided nothing new, minus - the count of the view the ladder actually produced. That is the right - comparand because what breaks the provider's cache is this view - differing from the last one -- not its distance from raw history. - - Both sides are `_estimate_tokens` over a full message list, i.e. the - same units the ladder itself uses, recomputed exactly rather than read - off the ladder's running counter (which carries small deliberate - approximations, e.g. the flat ~18-token stub charge at level 8). - No second estimator is introduced. - """ - pending = self._clear_at_least_pending - self._clear_at_least_pending = None - if pending is None: - # Predicate disabled, or this call made no new decision to judge. - return None - - required: int = pending["required"] - baseline_tokens: int = pending["baseline_tokens"] - freed = baseline_tokens - self._estimate_tokens(final_messages) - - if freed >= required: - self._clear_at_least_skips = 0 - logger.debug( - f"context-simple: clear_at_least ALLOWED this boundary -- " - f"level {max_level_reached} frees {freed:,} tokens " - f"(floor {required:,})" - ) - return None - - # --- REFUSED: roll the escalation back completely --- - self._clear_at_least_last_refused = True - self._clear_at_least_skips += 1 - self._restore_sticky_decisions(pending["sticky"]) - - baseline_view: list[dict[str, Any]] = pending["baseline_view"] - protected_tool_results_present = sum( - 1 for m in baseline_view if m.get("role") == "tool" - ) - protected_summary = ( - f"protected_tool_results={self.protected_tool_results} " - f"(of {protected_tool_results_present} tool results in the view), " - f"protected_recent={self.protected_recent:.0%} " - f"of {len(baseline_view)} messages" - ) - - if self._clear_at_least_skips >= self.compact_max_consecutive_skips: - # FAIL LOUD, do not degrade. Continuing to skip would let usage - # climb until the provider hard-fails with an opaque - # context-overflow error; silently compacting anyway would let a - # run "pass" while the predicate had quietly stopped applying. - # Neither tells the operator which knob to move -- this does. - raise RuntimeError( - f"context-simple: compact_clear_at_least={self.compact_clear_at_least!r} " - f"could not be satisfied {self._clear_at_least_skips} consecutive times " - f"(cap: {self.compact_max_consecutive_skips}). The compaction ladder " - f"reached level {max_level_reached} and freed only {freed:,} tokens " - f"against a required floor of {required:,}. Protected set holding the " - f"floor: {protected_summary}. Baseline view is {baseline_tokens:,} " - f"tokens against a budget of {pending['budget']:,}. Lower " - f"compact_clear_at_least, lower protected_recent/protected_tool_results, " - f"or raise the budget -- compacting harder will not help." - ) - - logger.info( - f"context-simple: clear_at_least REFUSED this boundary -- level " - f"{max_level_reached} would free only {freed:,} tokens against a floor of " - f"{required:,}; not worth a full prompt-cache rebuild. Escalation rolled " - f"back; returning the unchanged view " - f"({self._clear_at_least_skips}/{self.compact_max_consecutive_skips} " - f"consecutive skips)." - ) - - if self._hooks is not None: - try: - await self._hooks.emit( - "context:compaction-skipped", - { - "freed_tokens": freed, - "required_tokens": required, - "level_reached": max_level_reached, - "consecutive_skips": self._clear_at_least_skips, - "max_consecutive_skips": self.compact_max_consecutive_skips, - "baseline_tokens": baseline_tokens, - "budget": pending["budget"], - "protected_recent": self.protected_recent, - "protected_tool_results": self.protected_tool_results, - }, - ) - except Exception as e: # pragma: no cover - defensive - logger.warning(f"Could not emit compaction-skipped event: {e}") - - return baseline_view - async def _compact_ephemeral( self, budget: int, source_messages: list[dict[str, Any]] | None = None ) -> list[dict[str, Any]]: @@ -2077,13 +1755,6 @@ async def _compact_ephemeral( source_messages: Messages to compact. If None, uses self.messages. This allows compacting factory-generated message lists. """ - # The worth-the-rebuild predicate's comparand is strictly - # call-scoped. Clear it up front so a value left behind by an - # earlier call that raised can never be consumed by this one. - # A no-op while the predicate is disabled (it is never set). - self._clear_at_least_pending = None - self._clear_at_least_last_refused = False - messages_to_compact = ( source_messages if source_messages is not None else self.messages ) @@ -2240,38 +1911,6 @@ async def _compact_ephemeral( target_tokens, ) - # === WORTH-THE-REBUILD PREDICATE: capture the comparand === - # - # This is the last point at which nothing new has been decided. - # `working_messages` here is exactly what this call would return if - # it escalated no further (the sticky baseline), and `current_tokens` - # is exactly its token count -- `_estimate_tokens` is a per-message - # sum, so system_tokens + non_system_tokens IS the count of the - # combined list, not an approximation of it. - # - # Placed AFTER both early returns on purpose: a call where sticky - # state alone was already sufficient decided nothing, so there is - # nothing to judge and it must never count as a skip. The - # summary-swap-only path is likewise not judged here -- a swap has - # already mutated self.messages irreversibly by this point, so it - # cannot be rolled back; that path is governed by the summary shrink - # guard in _swap_in_pending_summary instead. - # - # The verdict itself runs in _finalize_compaction_with_stats, the - # single terminal choke point every escalation level returns through - # -- and BEFORE that method records stats or emits - # `context:compaction`, so a refused escalation never emits an event - # claiming a compaction that did not happen. - clear_at_least_required = self._clear_at_least_required(budget) - if clear_at_least_required > 0: - self._clear_at_least_pending = { - "required": clear_at_least_required, - "baseline_view": system_messages + list(working_messages), - "baseline_tokens": current_tokens, - "sticky": self._snapshot_sticky_decisions(), - "budget": budget, - } - logger.info( f"Compacting context (new escalation): {old_count} raw messages, {old_tokens:,} raw tokens " f"-> {len(working_messages)} messages, {current_tokens:,} tokens after sticky state " @@ -2970,17 +2609,6 @@ async def _finalize_compaction_with_stats( # System messages were extracted before compaction and must be restored final_messages = system_messages + working_messages - # === WORTH-THE-REBUILD PREDICATE: the verdict === - # Runs before ANY durable effect of this method (sticky level, - # stats, `context:compaction` event), so a refusal leaves no trace - # that a compaction occurred. Returns None -- and costs nothing but - # a None check -- while the predicate is disabled. - refused_view = await self._clear_at_least_verdict( - final_messages, max_level_reached - ) - if refused_view is not None: - return refused_view - final_tokens = self._estimate_tokens(final_messages) system_count = len(system_messages) tool_use_count = sum(1 for m in final_messages if m.get("tool_calls")) @@ -4024,52 +3652,11 @@ async def _swap_in_pending_summary( ) return non_system_messages, False - absorbed_messages = [ - msg for msg in non_system_messages if self._extract_seq(msg) in absorb_seqs - ] - summary_message = self._make_summary_message(pending["text"]) - - # === SHRINK GUARD (deepseek-harness compaction-basic, region.ts) === - # - # Refuse a summary that is not SMALLER than the span it replaces. Such - # a swap is a pure loss twice over: it pays a full cold prompt-cache - # rebuild (any shrink or edit invalidates the cached prefix) AND ends - # up with a bigger request than it started with -- a silent cost - # regression with no upside whatsoever. Nothing in this module - # previously checked it; a terse span plus a verbose 5-section summary - # is entirely capable of growing. - # - # Both sides are measured as full message dicts through the same - # `_estimate_tokens` the rest of the ladder uses, so the comparison is - # apples-to-apples (metadata overhead counted on both sides). - # - # There is no config knob for this: there is no defensible reason to - # want a summary that makes the context bigger. On refusal the pending - # summary is DISCARDED and this pass falls back to progressive - # compaction -- exactly the graceful path a stale summary already - # takes, not a failure. - # - # `_make_summary_message` has already consumed a `_seq` for the message - # being discarded. That gap is deliberately left rather than rewound: - # sequence ids need only be unique and monotonic (see - # _apply_sticky_decisions / _hybrid_split), and rewinding a shared - # counter is a worse hazard than an unused number. - absorbed_tokens = self._estimate_tokens(absorbed_messages) - summary_tokens = self._estimate_tokens([summary_message]) - if summary_tokens >= absorbed_tokens: - logger.warning( - f"context-simple: REFUSING summary swap -- the summary is " - f"{summary_tokens:,} tokens against the {absorbed_tokens:,} tokens " - f"of the {len(absorbed_messages)} messages it would replace, so it " - f"would GROW the context while still paying a full prompt-cache " - f"rebuild. Discarding it; falling back to progressive compaction " - f"for this pass." - ) - return non_system_messages, False - - for msg in absorbed_messages: - self._record_removed(msg) + for msg in non_system_messages: + if self._extract_seq(msg) in absorb_seqs: + self._record_removed(msg) + summary_message = self._make_summary_message(pending["text"]) self.messages.append(summary_message) self._summary_absorbed_count += len(absorb_seqs) diff --git a/tests/test_clear_at_least_predicate.py b/tests/test_clear_at_least_predicate.py deleted file mode 100644 index c12d520..0000000 --- a/tests/test_clear_at_least_predicate.py +++ /dev/null @@ -1,770 +0,0 @@ -"""Adversarial tests for the worth-the-rebuild predicate -(`compact_clear_at_least`) and the summary shrink guard. - -WHY THE PREDICATE EXISTS ------------------------- -The compaction trigger fires on a usage THRESHOLD and never asks how many -tokens the compaction will actually free. Every compaction shrinks the -request, and a shrink is a guaranteed cold prompt-cache rebuild on the OpenAI -path (the cache matches forward from a cached entry, never backward into one: -a strict byte-identical PREFIX of a cached request measured 0 cache_read). -So a boundary that frees very little pays a full rebuild of an ~18k-token -pinned head to buy almost nothing. - -`compact_clear_at_least` refuses those boundaries. It is Anthropic's -context-editing parameter of the same name, implemented client-side. - -WHAT THESE TESTS ARE ADVERSARIAL ABOUT --------------------------------------- -The predicate's own failure mode is STARVATION: refuse forever, usage climbs, -and the provider eventually hard-fails with an opaque context-overflow error. -So the tests below deliberately try to: - - 1. make it skip a call that decided nothing new (which must NOT count as a - skip -- otherwise a quiet session fails loud for no reason); - 2. make it starve silently (it must raise, naming the protected set); - 3. make it leave state behind after a refusal (sticky decisions, stats, the - tail notice, and the emitted events must all look as if the escalation - never happened); - 4. make the DEFAULT path behave differently from before the feature existed - (it must not -- byte-identical, no new events, predicate state never - even populated); - 5. split a tool_use/tool_result pair across a refusal. - -The shrink-guard tests do the same for a summary that would GROW the context: -a swap that pays a cache rebuild to make the request bigger is a pure loss -twice over, and nothing in this module previously checked for it. -""" - -from typing import Any - -import pytest -from amplifier_module_context_simple import SimpleContextManager - - -class _FakeHooks: - """Records every emitted event so tests can assert on the compaction / - compaction-skipped lifecycle without real HookRegistry internals.""" - - def __init__(self): - self.emitted: list[tuple[str, dict]] = [] - - async def emit(self, event, data): - self.emitted.append((event, data)) - - def names(self) -> list[str]: - return [name for name, _ in self.emitted] - - def payloads(self, event: str) -> list[dict]: - return [data for name, data in self.emitted if name == event] - - -def _padded(i: int, role: str, size: int = 80) -> dict: - """A message with enough bulk to move the token counter meaningfully.""" - return {"role": role, "content": f"{role} message {i} " + ("x" * size)} - - -def _tool_call(call_id: str, tool: str = "bash") -> dict: - return { - "role": "assistant", - "content": "", - "tool_calls": [{"id": call_id, "tool": tool, "arguments": {}}], - } - - -def _tool_result(call_id: str, content: str = "result") -> dict: - return {"role": "tool", "tool_call_id": call_id, "content": content} - - -def _make_context(**overrides: Any) -> SimpleContextManager: - """Same shape the sticky/notice suite uses, so these tests exercise the - real ladder rather than a bespoke configuration.""" - config: dict[str, Any] = { - "max_tokens": 2000, - "compact_threshold": 0.5, - "target_usage": 0.3, - "protected_recent": 0.2, - "protected_tool_results": 1, - "truncate_chars": 40, - "compaction_notice_enabled": True, - "compaction_notice_min_level": 1, - } - config.update(overrides) - return SimpleContextManager(**config) - - -async def _fill_until_compacted( - context: SimpleContextManager, turns: int = 40 -) -> None: - for i in range(turns): - await context.add_message(_padded(i, "user")) - await context.add_message(_padded(i, "assistant")) - - -def _normalize(messages: list[dict]) -> list[dict]: - """Drop the two fields that legitimately differ between two separately - built contexts (wall-clock `timestamp`) or between stored history and a - returned view (`_seq`, which `_finalize_view` strips at the module - boundary). Everything else must match exactly. - """ - out: list[dict] = [] - for msg in messages: - meta = { - k: v - for k, v in (msg.get("metadata") or {}).items() - if k not in ("timestamp", "_seq") - } - copy = {k: v for k, v in msg.items() if k != "metadata"} - if meta or "metadata" in msg: - copy["metadata"] = meta - out.append(copy) - return out - - -def _strip_ephemeral(messages: list[dict]) -> list[dict]: - """Drop the trailing ephemeral compaction notice, which is deliberately - outside the cached prefix and expected to vary.""" - out = list(messages) - while out and (out[-1].get("metadata") or {}).get("ephemeral"): - out.pop() - return out - - -# --------------------------------------------------------------------------- -# DEFAULT OFF: today's behaviour exactly -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_default_is_disabled_and_never_populates_predicate_state(): - """With no config, the predicate must not merely pass -- it must never be - evaluated at all. `_clear_at_least_pending` staying None is the proof that - the guarded code path is not entered, not just that it agreed.""" - context = _make_context() - assert context.compact_clear_at_least is None - await _fill_until_compacted(context) - - for _ in range(3): - await context.get_messages_for_request() - assert context._clear_at_least_pending is None - assert context._clear_at_least_skips == 0 - - assert context._last_compaction_stats is not None, ( - "setup must actually compact for this test to mean anything" - ) - - -@pytest.mark.asyncio -async def test_default_view_is_identical_to_explicit_zero_and_to_disabled(): - """`None` (default), `0`, and a negative value must all produce the exact - same returned view AND the exact same emitted events as each other.""" - views: list[list[dict]] = [] - event_names: list[list[str]] = [] - - for value in (None, 0, -5000): - hooks = _FakeHooks() - context = _make_context(compact_clear_at_least=value, hooks=hooks) - await _fill_until_compacted(context) - # Several calls, with history growing in between: exercises the sticky - # path and the escalation path, not just one call. - collected: list[dict] = [] - for i in range(3): - collected = await context.get_messages_for_request() - await context.add_message(_padded(5000 + i, "user")) - await context.add_message(_padded(5000 + i, "assistant")) - views.append(collected) - event_names.append(hooks.names()) - - assert _normalize(views[0]) == _normalize(views[1]) == _normalize(views[2]) - assert event_names[0] == event_names[1] == event_names[2] - assert "context:compaction-skipped" not in event_names[0] - - -@pytest.mark.asyncio -async def test_disabled_predicate_emits_no_skipped_event_ever(): - hooks = _FakeHooks() - context = _make_context(hooks=hooks) - await _fill_until_compacted(context) - for _ in range(5): - await context.get_messages_for_request() - await context.add_message(_padded(1, "user")) - assert "context:compaction-skipped" not in hooks.names() - assert "context:compaction" in hooks.names() - - -# --------------------------------------------------------------------------- -# BLOCK: a low-yield boundary is refused, and refused CLEANLY -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_predicate_blocks_a_low_yield_boundary(): - """An unreachable floor must refuse the boundary: the view comes back - uncompacted-by-this-call, and a `context:compaction-skipped` event carries - what was freed vs. what was required.""" - hooks = _FakeHooks() - context = _make_context(compact_clear_at_least=10_000_000, hooks=hooks) - await _fill_until_compacted(context) - - baseline_uncompacted = [ - m for m in context.messages if m.get("role") != "system" - ] - view = await context.get_messages_for_request() - - assert "context:compaction" not in hooks.names(), ( - "a refused escalation must not emit a compaction event -- no " - "compaction happened" - ) - skipped = hooks.payloads("context:compaction-skipped") - assert len(skipped) == 1 - assert skipped[0]["required_tokens"] == 10_000_000 - assert skipped[0]["freed_tokens"] < 10_000_000 - assert skipped[0]["consecutive_skips"] == 1 - assert skipped[0]["level_reached"] >= 1, ( - "the ladder must actually have run -- the predicate judges a real " - "result, not a guess made before doing the work" - ) - - # The returned view is the untouched history (no truncation, no removal). - assert _normalize(_strip_ephemeral(view)) == _normalize(baseline_uncompacted) - - -@pytest.mark.asyncio -async def test_refusal_rolls_back_every_sticky_decision(): - """The escalation's truncate/remove/stub decisions are the only durable - state written before the predicate runs. A refusal must undo all three -- - otherwise the NEXT call silently replays a compaction that was refused.""" - context = _make_context(compact_clear_at_least=10_000_000) - await _fill_until_compacted(context) - - before = ( - set(context._removed_seqs), - set(context._truncated_seqs), - set(context._stubbed_seqs), - context._sticky_level, - ) - await context.get_messages_for_request() - after = ( - set(context._removed_seqs), - set(context._truncated_seqs), - set(context._stubbed_seqs), - context._sticky_level, - ) - assert before == after - assert context._last_compaction_stats is None, ( - "a refused escalation must leave no stats behind -- otherwise the " - "tail notice would announce a compaction that never happened" - ) - - -@pytest.mark.asyncio -async def test_refusal_leaves_no_compaction_notice(): - context = _make_context(compact_clear_at_least=10_000_000) - await _fill_until_compacted(context) - view = await context.get_messages_for_request() - assert not any( - (m.get("metadata") or {}).get("source") == "context-compaction" - for m in view - ) - - -@pytest.mark.asyncio -async def test_refusal_preserves_tool_pair_integrity(): - """The refused view is raw history, so every tool_calls message must still - be answered by its tool result. A refusal must never be able to strand a - tool_use without its tool_result.""" - context = _make_context(compact_clear_at_least=10_000_000) - for i in range(30): - await context.add_message(_padded(i, "user")) - await context.add_message(_tool_call(f"call-{i}")) - await context.add_message(_tool_result(f"call-{i}", "r" * 200)) - - view = await context.get_messages_for_request() - - called = [ - tc["id"] - for m in view - for tc in (m.get("tool_calls") or []) - ] - answered = [ - m["tool_call_id"] for m in view if m.get("role") == "tool" - ] - assert called, "setup must produce tool calls" - assert sorted(called) == sorted(answered) - - -# --------------------------------------------------------------------------- -# ALLOW: a high-yield boundary passes, byte-identically to disabled -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_predicate_allows_a_high_yield_boundary(): - hooks = _FakeHooks() - context = _make_context(compact_clear_at_least=1, hooks=hooks) - await _fill_until_compacted(context) - await context.get_messages_for_request() - - assert "context:compaction" in hooks.names() - assert "context:compaction-skipped" not in hooks.names() - assert context._clear_at_least_skips == 0 - assert context._last_compaction_stats is not None - - -@pytest.mark.asyncio -async def test_allowed_boundary_is_identical_to_the_disabled_path(): - """A floor of 1 token is satisfiable by any real compaction, so the - resulting view must be byte-identical to running with the predicate off. - This is what proves the predicate ALLOWS rather than perturbs.""" - allowed = _make_context(compact_clear_at_least=1) - disabled = _make_context() - for ctx in (allowed, disabled): - await _fill_until_compacted(ctx) - - for _ in range(3): - a = await allowed.get_messages_for_request() - d = await disabled.get_messages_for_request() - assert _normalize(a) == _normalize(d) - for ctx in (allowed, disabled): - await ctx.add_message(_padded(7001, "user")) - await ctx.add_message(_padded(7001, "assistant")) - - -@pytest.mark.asyncio -async def test_freed_is_marginal_not_measured_against_raw_history(): - """The predicate must judge the MARGINAL reclaim of this boundary, not the - distance from raw history. If it used raw history, an already-compacted - session would report a huge (stale) "freed" every call and the predicate - would approve boundaries that free nothing. - - Here: allow the first escalation with a low floor, then raise the floor - beyond anything a later marginal escalation can free, and assert the later - call is judged on the marginal number. - """ - hooks = _FakeHooks() - context = _make_context(compact_clear_at_least=1, hooks=hooks) - await _fill_until_compacted(context) - await context.get_messages_for_request() - assert "context:compaction" in hooks.names() - - stats = context._last_compaction_stats - assert stats is not None - already_freed = stats["before_tokens"] - stats["after_tokens"] - assert already_freed > 0 - - # Now require MORE than the whole first compaction freed. Any further - # marginal escalation frees far less than that, so it must be refused. - context.compact_clear_at_least = already_freed + 100_000 - for i in range(20): - await context.add_message(_padded(8000 + i, "user")) - await context.add_message(_padded(8000 + i, "assistant")) - await context.get_messages_for_request() - - skipped = hooks.payloads("context:compaction-skipped") - assert skipped, ( - "the second boundary must be judged on its own marginal reclaim, " - "which is far below the raw-history delta" - ) - assert skipped[-1]["freed_tokens"] < already_freed - - -# --------------------------------------------------------------------------- -# STARVATION: the predicate's own failure mode -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_no_op_calls_never_count_as_skips(): - """THE trap. Once sticky state alone keeps the view under threshold, the - ladder decides nothing new and returns early. Those calls must NOT be - judged: nothing was refused, so counting them would fail loud on a session - that is behaving perfectly.""" - context = _make_context(compact_clear_at_least=1, compact_max_consecutive_skips=2) - await _fill_until_compacted(context) - await context.get_messages_for_request() # a real, allowed escalation - - # Many further calls with no growth: sticky state alone suffices. - for _ in range(10): - await context.get_messages_for_request() - - assert context._clear_at_least_skips == 0 - - -@pytest.mark.asyncio -async def test_fails_loud_after_max_consecutive_skips(): - """Silent starvation is the one outcome this must never produce. After the - cap it raises, naming what was freed, what was required, and the protected - set that is holding the floor -- i.e. which knob to move.""" - context = _make_context( - compact_clear_at_least=10_000_000, compact_max_consecutive_skips=3 - ) - await _fill_until_compacted(context) - - await context.get_messages_for_request() - assert context._clear_at_least_skips == 1 - await context.get_messages_for_request() - assert context._clear_at_least_skips == 2 - - with pytest.raises(RuntimeError) as exc: - await context.get_messages_for_request() - - message = str(exc.value) - assert "compact_clear_at_least" in message - assert "10000000" in message.replace(",", "") - assert "protected_recent" in message - assert "protected_tool_results" in message - assert "freed only" in message - - -@pytest.mark.asyncio -async def test_fail_loud_still_rolls_back_state_before_raising(): - """Even on the fatal path the escalation is undone first, so a caller that - catches the error is not left with half-applied compaction decisions.""" - context = _make_context( - compact_clear_at_least=10_000_000, compact_max_consecutive_skips=1 - ) - await _fill_until_compacted(context) - before = set(context._removed_seqs), set(context._truncated_seqs) - - with pytest.raises(RuntimeError): - await context.get_messages_for_request() - - assert (set(context._removed_seqs), set(context._truncated_seqs)) == before - assert context._clear_at_least_pending is None, ( - "call-scoped state must never leak past a raise" - ) - - -@pytest.mark.asyncio -async def test_skip_streak_resets_after_an_accepted_compaction(): - context = _make_context( - compact_clear_at_least=10_000_000, compact_max_consecutive_skips=5 - ) - await _fill_until_compacted(context) - await context.get_messages_for_request() - await context.get_messages_for_request() - assert context._clear_at_least_skips == 2 - - context.compact_clear_at_least = 1 - await context.get_messages_for_request() - assert context._clear_at_least_skips == 0 - - -@pytest.mark.asyncio -async def test_max_consecutive_skips_zero_is_clamped_to_one_not_to_infinity(): - """A cap of 0 read literally means "tolerate unlimited refusals" -- which - is exactly the silent hang the cap exists to prevent. It must clamp UP to - 1, never be honoured as "never fail".""" - context = _make_context( - compact_clear_at_least=10_000_000, compact_max_consecutive_skips=0 - ) - assert context.compact_max_consecutive_skips == 1 - await _fill_until_compacted(context) - with pytest.raises(RuntimeError): - await context.get_messages_for_request() - - -@pytest.mark.asyncio -async def test_skip_streak_does_not_survive_clear_or_resume(): - """A streak accumulated against one message set says nothing about a - different one; inheriting it would fail loud on the first refusal after a - resume.""" - context = _make_context( - compact_clear_at_least=10_000_000, compact_max_consecutive_skips=3 - ) - await _fill_until_compacted(context) - await context.get_messages_for_request() - assert context._clear_at_least_skips == 1 - - await context.clear() - assert context._clear_at_least_skips == 0 - - await _fill_until_compacted(context) - await context.get_messages_for_request() - assert context._clear_at_least_skips == 1 - - await context.set_messages([_padded(i, "user") for i in range(3)]) - assert context._clear_at_least_skips == 0 - assert context._clear_at_least_pending is None - - -# --------------------------------------------------------------------------- -# PREFIX / _seq STABILITY across a refusal -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_refusal_is_prefix_stable_and_append_only(): - """Two consecutive refused calls with one turn of growth in between must - share a byte-identical prefix -- the whole point of refusing is to keep - the cached prefix intact, so a refusal that reshuffled the view would be - worse than compacting.""" - context = _make_context( - compact_clear_at_least=10_000_000, compact_max_consecutive_skips=99 - ) - await _fill_until_compacted(context) - - first = _strip_ephemeral(await context.get_messages_for_request()) - await context.add_message(_padded(9001, "user")) - await context.add_message(_padded(9001, "assistant")) - second = _strip_ephemeral(await context.get_messages_for_request()) - - assert len(second) == len(first) + 2 - assert second[: len(first)] == first, ( # exact: same context, no normalisation - "a refusal must be strictly append-only: the shared prefix cannot move" - ) - - -@pytest.mark.asyncio -async def test_seq_identity_is_untouched_by_a_refusal(): - """`_seq` is compaction identity. A refusal must not renumber, drop, or - duplicate any of them.""" - context = _make_context( - compact_clear_at_least=10_000_000, compact_max_consecutive_skips=99 - ) - await _fill_until_compacted(context) - before = [m["metadata"]["_seq"] for m in context.messages] - - for _ in range(3): - await context.get_messages_for_request() - - after = [m["metadata"]["_seq"] for m in context.messages] - assert before == after - assert len(set(after)) == len(after) - - -@pytest.mark.asyncio -async def test_refusal_then_acceptance_still_compacts_correctly(): - """A refusal must not poison the next real escalation: once the floor is - satisfiable, compaction proceeds and produces a genuinely smaller view.""" - context = _make_context( - compact_clear_at_least=10_000_000, compact_max_consecutive_skips=99 - ) - await _fill_until_compacted(context) - refused = await context.get_messages_for_request() - - context.compact_clear_at_least = 1 - accepted = await context.get_messages_for_request() - - assert context._estimate_tokens(accepted) < context._estimate_tokens(refused) - assert context._last_compaction_stats is not None - - -# --------------------------------------------------------------------------- -# Floor resolution: absolute vs fraction, and malformed values -# --------------------------------------------------------------------------- - - -@pytest.mark.parametrize( - "raw, budget, expected", - [ - (None, 100_000, 0), - (0, 100_000, 0), - (-1, 100_000, 0), - (-0.5, 100_000, 0), - (20_000, 100_000, 20_000), - (1, 100_000, 1), - (0.25, 100_000, 25_000), - (0.25, 0, 0), - (0.999, 1_000, 999), - (1.0, 100_000, 1), - ("nonsense", 100_000, 0), - (object(), 100_000, 0), - ], -) -def test_floor_resolution(raw, budget, expected): - """A float in (0, 1) is a FRACTION of the budget -- a hardcoded token floor - silently means something different on a 200k window than a 45k one. - - 1.0 is deliberately ABSOLUTE (1 token), not "100% of budget": a floor of - one whole budget can never be met, so reading it as a fraction would turn - a plausible-looking config into a guaranteed fail-loud. - """ - context = _make_context(compact_clear_at_least=raw) - assert context._clear_at_least_required(budget) == expected - - -@pytest.mark.asyncio -async def test_malformed_floor_disables_rather_than_crashing_a_session(): - """Consistent with how this module handles every other unrecognized config - value: warn and fall back, never take a session down on config alone.""" - context = _make_context(compact_clear_at_least="twenty thousand") - await _fill_until_compacted(context) - view = await context.get_messages_for_request() - assert view - assert context._last_compaction_stats is not None - assert context._clear_at_least_skips == 0 - - -@pytest.mark.asyncio -async def test_fraction_floor_scales_with_the_budget(): - """Same fraction, two budgets: the small-budget session's boundary clears - a floor the large-budget session's does not.""" - small = _make_context(max_tokens=2000, compact_clear_at_least=0.05) - assert small._clear_at_least_required(2000) == 100 - assert small._clear_at_least_required(200_000) == 10_000 - - -# --------------------------------------------------------------------------- -# Summary shrink guard -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_shrink_guard_refuses_a_summary_larger_than_what_it_replaces(): - """A summary that GROWS the context is a pure loss twice over: it pays a - full cache rebuild AND ends up with a bigger request. Refuse it.""" - context = SimpleContextManager(compaction_strategy="summary", protected_recent=0.9) - for i in range(5): - await context.add_message({"role": "user", "content": f"t{i}"}) - - seqs = [m["metadata"]["_seq"] for m in context.messages[:2]] - context._pending_summary = {"seqs": frozenset(seqs), "text": "V" * 20_000} - - non_system = [m for m in context.messages if m.get("role") != "system"] - messages_before = list(context.messages) - - result, did_swap = await context._swap_in_pending_summary(non_system) - - assert did_swap is False - assert result == non_system - assert context.messages == messages_before, ( - "a refused summary must never be appended to history" - ) - assert context._pending_summary is None - assert context._summary_absorbed_count == 0 - assert context._removed_seqs == set(), ( - "a refused summary must not record its span as removed" - ) - assert context._summarization_failures == 0, ( - "refusing an unprofitable summary is a graceful fallback, not a " - "summarizer failure" - ) - - -@pytest.mark.asyncio -async def test_shrink_guard_allows_a_summary_that_is_genuinely_smaller(): - """The guard must not block the case the feature exists for.""" - context = SimpleContextManager(compaction_strategy="summary", protected_recent=0.9) - for i in range(5): - await context.add_message({"role": "user", "content": f"turn {i} " + "x" * 400}) - - seqs = [m["metadata"]["_seq"] for m in context.messages[:2]] - context._pending_summary = {"seqs": frozenset(seqs), "text": "tiny summary"} - - non_system = [m for m in context.messages if m.get("role") != "system"] - result, did_swap = await context._swap_in_pending_summary(non_system) - - assert did_swap is True - assert context._summary_absorbed_count == 2 - assert set(seqs) <= context._removed_seqs - assert result[-1]["metadata"]["type"] == "context_summary" - - -@pytest.mark.asyncio -async def test_shrink_guard_refuses_an_equal_sized_summary(): - """"Not smaller" includes "exactly the same size": paying a cache rebuild - to swap content for content of identical cost buys nothing.""" - context = SimpleContextManager(compaction_strategy="summary", protected_recent=0.9) - await context.add_message({"role": "user", "content": "x"}) - - seqs = [context.messages[0]["metadata"]["_seq"]] - absorbed_tokens = context._estimate_tokens(context.messages[:1]) - - # Binary-search a summary text whose full message dict prices at exactly - # the absorbed span's estimate, so the >= boundary itself is exercised. - text = "" - while True: - probe_seq = context._next_seq - probe = context._make_summary_message(text) - context._next_seq = probe_seq # probe only; do not consume the id - if context._estimate_tokens([probe]) >= absorbed_tokens: - break - text += "y" - - context._pending_summary = {"seqs": frozenset(seqs), "text": text} - non_system = list(context.messages) - _result, did_swap = await context._swap_in_pending_summary(non_system) - assert did_swap is False - - -@pytest.mark.asyncio -async def test_progressive_mode_never_reaches_the_shrink_guard(): - """The guard lives on the summary path only; the default mode must be - untouched by it.""" - context = _make_context() - await _fill_until_compacted(context) - await context.get_messages_for_request() - assert context._pending_summary is None - assert context._summary_absorbed_count == 0 - assert context._last_compaction_stats is not None - assert "messages_absorbed_by_summary" not in context._last_compaction_stats - - -# --------------------------------------------------------------------------- -# Composition -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_predicate_composes_with_hybrid_token_meter(): - """The predicate acts on the ladder's own estimator units, so it must not - break when the TRIGGER is driven by a provider-anchored count instead.""" - context = _make_context(compact_clear_at_least=1, token_meter="hybrid") - await _fill_until_compacted(context) - view = await context.get_messages_for_request() - assert view - - -@pytest.mark.asyncio -async def test_predicate_survives_a_context_with_no_hooks(): - """Event emission is optional; refusing must not depend on it.""" - context = _make_context( - compact_clear_at_least=10_000_000, compact_max_consecutive_skips=99, hooks=None - ) - await _fill_until_compacted(context) - view = await context.get_messages_for_request() - assert view - assert context._clear_at_least_skips == 1 - - -@pytest.mark.asyncio -async def test_a_refusal_is_not_a_boundary_for_the_last_user_replay(): - """`replay_last_user_on_compaction` fires once per compaction BOUNDARY, - identified by (sticky_level, summary_absorbed_count). A refusal leaves - both unchanged by design -- so on the first-ever refusal that identity - still differs from the initial `None` and would look like a fresh - boundary. Appending a verbatim user replay after a compaction that did - not happen would both mislead the model and spend the tokens the refusal - exists to save. - """ - context = _make_context( - compact_clear_at_least=10_000_000, - compact_max_consecutive_skips=99, - replay_last_user_on_compaction=True, - ) - await _fill_until_compacted(context) - view = await context.get_messages_for_request() - - assert context._clear_at_least_last_refused is True - assert not any( - (m.get("metadata") or {}).get("source") == "context-replay" - for m in view - ), "a refused compaction must not mark or replay a boundary" - assert context._last_replayed_boundary is None - - -@pytest.mark.asyncio -async def test_replay_still_fires_on_an_allowed_boundary(): - """The guard above must not disable the feature it is protecting.""" - context = _make_context( - compact_clear_at_least=1, replay_last_user_on_compaction=True - ) - await _fill_until_compacted(context) - view = await context.get_messages_for_request() - - assert context._clear_at_least_last_refused is False - assert any( - (m.get("metadata") or {}).get("source") == "context-replay" - for m in view - ) From d5605e740c7e19ff14954cdf87d77531fadae839 Mon Sep 17 00:00:00 2001 From: bkrabach <702425+bkrabach@users.noreply.github.com> Date: Wed, 2 Sep 2026 17:08:02 -0700 Subject: [PATCH 4/7] Revert "feat: replay_last_user_on_compaction -- opt-in last-user replay at the tail (#4i3) (#24)" This reverts commit 3972070d7c368a844288a86787fa2935ba2391ae. Merge policy: main carries wins only. replay_last_user_on_compaction was an unproven, opt-in feature. Belongs on a branch for evaluation, not on main. --- README.md | 83 --- amplifier_module_context_simple/__init__.py | 277 -------- tests/test_replay_last_user.py | 702 -------------------- 3 files changed, 1062 deletions(-) delete mode 100644 tests/test_replay_last_user.py diff --git a/README.md b/README.md index fc62917..72ae0a8 100644 --- a/README.md +++ b/README.md @@ -638,89 +638,6 @@ computed by the new code. is the seam for it. - **A cleanup sweep.** See above. -## Last-user replay (`replay_last_user_on_compaction`) - -**Opt-in. Default `false`. Default is byte-identical to the behavior -before this feature existed. NOT MEASURED — see "Status" below.** - -```toml -config = { replay_last_user_on_compaction = true } -``` - -### What it does - -The highest-value tier of retained context is the **user's own -verbatims**. A compaction boundary can leave the most recent user -instruction sitting far from the attention-strongest tail position, -behind whatever tool results the ladder chose to keep. - -When this flag is on **and a compaction boundary actually occurs**, the -module appends a copy of the most recent real user message as the last -item before the dynamic tail (the compaction notice), wrapped in a -`` envelope that states -explicitly that it is a repeat and not a new request: - -``` -[... compacted conversation ...] -[replay ] user, ephemeral: … -[notice ] user, ephemeral: … -``` - -### Why it is shaped this way - -- **Append-only.** Nothing before the append point moves. No `_seq` is - consumed, no sticky decision is touched, `self.messages` is never - modified, and the message is ephemeral (view-only). Append is the - measured cache-**HIT** shape; a shrink or a reorder is a cold rebuild. -- **Both tail items are `ephemeral: true`,** so the Anthropic provider's - trailing-ephemeral walk-back skips them and the cache breakpoint lands - in the same place it did before. -- **Tool-pair integrity is untouched.** The replay applies the same - unanswered-`tool_calls` tail guard the compaction notice uses, and - skips rather than interleaving between a `tool_use` and its - `tool_result`. -- **Once per boundary, not once per request.** The boundary identity is - `(sticky progressive level, summary-absorbed count)`; a request that - merely re-applies an existing sticky decision does not re-emit. - -### When it deliberately does nothing - -| Condition | Why | -|---|---| -| No compaction this request | No boundary, nothing was buried | -| This boundary's replay already went out | Once per boundary | -| View ends on unanswered `tool_calls` | Tool-pair atomicity; retries next request | -| The last real user message is already the tail | A copy would be a pure duplicate | -| The message is a Level-8 stub | A stub is not the user's words | -| No real user message, or no text in it | Nothing to repeat | -| The copy would exceed the budget | Compaction just shed tokens to reach it | - -### `_is_real_user_message` is vendored, not imported - -This module declares **no runtime dependencies**, so the predicate is -implemented locally rather than imported from `amplifier-foundation` — a -soft import would make behavior depend on whether an undeclared package -happens to be installed. - -It is also deliberately **stronger** than foundation's. Measured against -`amplifier-foundation` 1.0.0 (`session/messages.py`): foundation rejects -only content beginning with the **bare** `` tag, so an -**attributed** envelope — including this module's own -`` summary message — passes -foundation's check as a real user turn. Replaying a synthetic envelope as -if it were the user's words is exactly what this feature must never do, -so the local predicate rejects any `=1.0.0` diff --git a/amplifier_module_context_simple/__init__.py b/amplifier_module_context_simple/__init__.py index 919b07c..77ad0c7 100644 --- a/amplifier_module_context_simple/__init__.py +++ b/amplifier_module_context_simple/__init__.py @@ -142,37 +142,6 @@ mutate an already-cached prefix -- which, under a grow-only prompt cache, is a full cold rebuild. A failed write yields a pointer to a missing file (visible, recoverable) rather than a silent byte change. - -Last-User Replay After a Compaction Boundary (opt-in, default off -- see -config `replay_last_user_on_compaction`): - • MOTIVATION (design lens): the highest-value retained tier is the - USER'S OWN VERBATIMS. A compaction boundary can leave the most recent - user instruction sitting far from the attention-strongest tail - position, behind a wall of tool results the ladder chose to keep. - • When enabled AND a compaction boundary actually occurs, this APPENDS - (never moves, never removes) a copy of the most recent real user - message as the last item before the dynamic tail (the compaction - notice), wrapped in a `` - envelope so it reads as a reminder of a standing instruction and is - never mistaken for a NEW request. - • APPEND-ONLY BY CONSTRUCTION -- the measured cache-HIT shape. Nothing - before the append point moves, no `_seq` is consumed, no sticky - decision is touched, and `self.messages` is never modified. The - replay is ephemeral (metadata.ephemeral=True), so it joins the - compaction notice in the Anthropic provider's trailing-ephemeral - exclusion walk and does not displace the cache breakpoint. - • Tool-pair integrity untouched: it uses the SAME tail guard as the - compaction notice and skips entirely when the view ends on an - assistant message with unanswered tool_calls. - • FIRES ONCE PER BOUNDARY, not once per request: the boundary identity - is (sticky progressive level, summary-absorbed count), so a request - that merely re-applies an existing sticky decision does not re-emit. - • NOT MEASURED. This is a mechanism, shipped default-off, with NO - quality evidence behind it: the retention scenario that could - discriminate (S7) does not exist yet, and S5-CRAC is saturated - (40/40 constraints, 20/20 post-compaction in every arm of every - probe 1-6). Do not enable by default, and do not claim a quality - benefit, until a discriminating eval has run. """ # Amplifier module metadata @@ -253,70 +222,6 @@ _SUMMARY_ENVELOPE_SOURCE = "context-summary" _SUMMARY_METADATA_TYPE = "context_summary" -# The last-user-replay envelope's source tag and metadata source marker -# (opt-in `replay_last_user_on_compaction`; see module docstring -# "Last-User Replay After a Compaction Boundary"). -_REPLAY_ENVELOPE_SOURCE = "context-replay" - -# Any `` opener, attributed or bare. See -# _is_real_user_message for why the bare-tag check is not sufficient. -_REMINDER_ENVELOPE_PREFIX = " bool: - """Whether `entry` is a genuine human-authored user turn. - - Mirrors `amplifier_foundation.session.is_real_user_message` (role is - "user", no `tool_call_id`, content not a reminder envelope), plus one - hardening clause. - - VENDORED, NOT IMPORTED, on purpose: this module declares no runtime - dependencies (pyproject `dependencies = []`). A soft import of - foundation would make this predicate's behavior depend on whether an - undeclared package happens to be installed in the host environment -- - two silently different code paths for the same config. Instead the - logic lives here and `tests/test_replay_last_user.py` asserts parity - against the real foundation function whenever foundation IS - importable, so drift is caught by a failing test rather than assumed - away. - - HARDENING, and it is load-bearing: foundation 1.0.0 - (session/messages.py:95, 103) rejects only content beginning with the - BARE `` tag. An ATTRIBUTED envelope -- including this - module's own `` summary - message and the `context-replay` message this feature emits -- does - NOT match that check and is classified by foundation as a real user - turn. (The comment at `_SUMMARY_ENVELOPE_SOURCE` and the docstring of - `_make_summary_message` both assert the opposite; they are wrong on - this point as of foundation 1.0.0.) Replaying a synthetic envelope as - if it were the user's own words -- or worse, replaying a replay -- is - precisely the failure this feature must never produce, so this - predicate rejects any content opening with ` None: self._stubbed_seqs = set() self._sticky_level = 0 self._last_compaction_stats = None - self._last_replayed_boundary = None self._reset_summary_strategy_state() logger.info(f"Restored {len(messages)} messages to context") @@ -1213,7 +1078,6 @@ async def clear(self) -> None: # forked session may still hold a pointer to one; see README). self._tool_name_by_call_id = {} self._spilled_paths = set() - self._last_replayed_boundary = None self._reset_summary_strategy_state() logger.info("Context cleared") @@ -3086,147 +2950,6 @@ def _format_affected_items(self, level: int, stats: dict[str, Any]) -> str: "- If context is critical, consider asking user to clarify their current goal" ) - # ------------------------------------------------------------------ - # Last-user replay (`replay_last_user_on_compaction: true`) - # ------------------------------------------------------------------ - # - # Opt-in, default off, and a complete no-op when off: the ONLY call - # site is guarded by `if self.replay_last_user_on_compaction` inside - # the compaction branch of get_messages_for_request(), so in the - # default configuration none of the code below ever executes and the - # returned view is byte-identical to before this feature existed. - - def _current_compaction_boundary(self) -> tuple[int, int]: - """Identity of the compaction boundary the view currently reflects. - - A "boundary" is a real escalation of what has been shed, not a - request: the progressive ladder's cumulative sticky level, paired - with how many messages the summary strategy has absorbed. Both - are monotonic within a session and both already exist -- this - introduces no new state to keep in sync. - - A request that merely re-applies existing sticky decisions leaves - both components unchanged, which is exactly what makes the replay - fire once per boundary instead of once per request. - """ - return (self._sticky_level, self._summary_absorbed_count) - - @staticmethod - def _replay_text(msg: dict[str, Any]) -> str: - """Plain text of a user message, for string and block-list content. - - Returns "" when there is no text to replay (e.g. an image-only - block list), which the caller treats as "nothing to say, skip". - """ - content = msg.get("content") - if isinstance(content, str): - return content - if isinstance(content, list): - parts = [ - block["text"] - for block in content - if isinstance(block, dict) and isinstance(block.get("text"), str) - ] - return "\n".join(part for part in parts if part) - return "" - - def _maybe_append_last_user_replay( - self, compacted: list[dict[str, Any]], budget: int - ) -> None: - """Append one reminder-wrapped copy of the most recent real user - message to the tail of `compacted`, in place, if all guards pass. - - Guards, and why each exists: - - 1. ONCE PER BOUNDARY -- skip when this boundary's replay already - went out (see _current_compaction_boundary). - 2. TOOL-PAIR ATOMICITY -- skip when the view ends on an assistant - message with unanswered tool_calls, for exactly the reason the - compaction notice does: a user-role message landing between a - tool_use and its tool_result is rejected or mishandled by - providers. Skipping is free; the boundary stays unmarked and - the replay goes out on the next request instead. - 3. NOT ALREADY THE TAIL -- if the last real user message IS the - final item, a copy would be a pure duplicate that says nothing - the model cannot already see in the strongest position. - 4. NOTHING TO SAY -- no real user message in view, or no text in - it. - 5. NOT A STUB -- a Level-8 stub (`_stubbed`) is not the user's - words, it is a placeholder REPLACING them. The envelope below - calls its payload "a verbatim copy of your most recent - instruction"; emitting a stub under that sentence would make - the envelope lie. DEFENCE IN DEPTH, and deliberately so: this - branch is UNREACHABLE as of this commit, because the ladder - protects the last user message from stubbing at every level - (`i != last_user_idx` in _remove_messages_with_protection, and - `first_user_idx != last_user_idx` at Level 8). The guard exists - so this feature's honesty does not silently depend on a - protection rule enforced 600 lines away in code it does not - own. tests/test_replay_last_user.py exercises it directly - rather than pretending a natural fixture reaches it. - 6. BUDGET -- compaction has just finished shedding tokens to hit - `budget`; a large pasted user message (a file, a log) could - push the request straight back over it. Skip rather than - overshoot what compaction just paid for. - - Sourced from the COMPACTED VIEW, not from self.messages, on - purpose: the view is post-sticky-decision, so this can never - resurrect content compaction deliberately shed (e.g. a Level-8 - stub of a first-and-only user message replays as the stub, not as - the original text it replaced). - - Mutates only the caller's local view list. self.messages, `_seq` - allocation, and every sticky decision set are untouched. - """ - boundary = self._current_compaction_boundary() - if boundary == self._last_replayed_boundary: - return - - if not compacted or compacted[-1].get("tool_calls"): - return - - source = next( - (msg for msg in reversed(compacted) if _is_real_user_message(msg)), None - ) - if source is None or source is compacted[-1] or source.get("_stubbed"): - return - - text = self._replay_text(source) - if not text.strip(): - return - - replay = { - "role": "user", - "content": ( - f'\n' - "This is a verbatim copy of your most recent instruction, " - "repeated here because context compaction has moved it far " - "from the end of the conversation. It is NOT a new request " - "-- do not answer it again if it is already handled.\n\n" - f"{text}\n" - "" - ), - "metadata": { - "source": _REPLAY_ENVELOPE_SOURCE, - "ephemeral": True, - }, - } - - projected = self._estimate_tokens(compacted) + self._estimate_tokens([replay]) - if budget > 0 and projected > budget: - logger.debug( - "Skipping last-user replay this request: it would put the " - f"view at ~{projected:,} tokens against a {budget:,} budget " - "that compaction just shed tokens to reach" - ) - return - - compacted.append(replay) - self._last_replayed_boundary = boundary - logger.debug( - f"Appended last-user replay at tail for compaction boundary {boundary}" - ) - # ------------------------------------------------------------------ # Summary compaction strategy (`compaction_strategy: "summary"`) # ------------------------------------------------------------------ diff --git a/tests/test_replay_last_user.py b/tests/test_replay_last_user.py deleted file mode 100644 index 2e88fde..0000000 --- a/tests/test_replay_last_user.py +++ /dev/null @@ -1,702 +0,0 @@ -""" -Tests for `replay_last_user_on_compaction` -- appending a reminder-wrapped -copy of the most recent real user message at the tail once per compaction -boundary. - -The load-bearing tests here, in order of what they protect: - - 1. `test_default_off_is_byte_identical` -- the default configuration must - produce a view byte-identical to one built by a manager that has never - heard of this feature. This is the test that keeps an opt-in flag - honest. - 2. `test_fires_once_per_boundary` -- the replay is keyed to a compaction - BOUNDARY, not to a request. A request that merely re-applies existing - sticky decisions must not re-emit it. - 3. `test_prefix_stability_with_replay_enabled` -- enabling the feature - must not disturb the byte-stable shared prefix the sticky/_seq - machinery exists to protect. - 4. `test_no_duplicate_when_last_message_is_already_the_target` -- a copy - of the tail, at the tail, is pure waste. - 5. `test_never_replays_a_reminder_envelope` / - `test_predicate_matches_foundation_where_foundation_rejects` -- the - replay must carry the USER's words, never a synthetic envelope - (including one of its own). -""" - -import copy -from typing import Any - -import pytest -from amplifier_module_context_simple import ( - _REPLAY_ENVELOPE_SOURCE, - SimpleContextManager, - _is_real_user_message, -) - -BASE_CONFIG: dict[str, Any] = { - "max_tokens": 2000, - "compact_threshold": 0.5, - "target_usage": 0.3, - "protected_recent": 0.2, - "protected_tool_results": 1, - "truncate_chars": 40, - "compaction_notice_enabled": True, - "compaction_notice_min_level": 1, -} - - -def _make_context(**overrides: Any) -> SimpleContextManager: - config = dict(BASE_CONFIG) - config.update(overrides) - return SimpleContextManager(**config) - - -def _padded(i: int, role: str, size: int = 80) -> dict[str, Any]: - return {"role": role, "content": f"{role} message {i} " + ("x" * size)} - - -async def _fill_until_compacted( - context: SimpleContextManager, turns: int = 40 -) -> None: - for i in range(turns): - await context.add_message(_padded(i, "user")) - await context.add_message(_padded(i, "assistant")) - - -def _replays(messages: list[dict[str, Any]]) -> list[dict[str, Any]]: - return [ - m - for m in messages - if (m.get("metadata") or {}).get("source") == _REPLAY_ENVELOPE_SOURCE - ] - - -def _without_timestamps(messages: list[dict[str, Any]]) -> list[dict[str, Any]]: - """Drop the wall-clock timestamp add_message() stamps on every message. - - Two managers fed the same history milliseconds apart get different - timestamps; that difference is pre-existing behavior and has nothing to - do with this feature. Everything else -- roles, content, ordering, every - other metadata key -- is compared byte for byte. - """ - scrubbed = [] - for msg in messages: - meta = msg.get("metadata") - if isinstance(meta, dict) and "timestamp" in meta: - msg = {**msg, "metadata": {k: v for k, v in meta.items() if k != "timestamp"}} - scrubbed.append(msg) - return scrubbed - - -# An ESCALATING fixture: unlike flat text turns (which jump straight to -# level 8 on the first compaction), tool-result turns give the truncation -# levels real work to do, so the sticky level climbs in observable steps. -# Measured with this exact configuration: level 0 through batch 3, level 3 -# at batch 4, unchanged through batch 7, level 5 at batch 8. That gives a -# deterministic "same boundary" window AND a deterministic escalation. -ESCALATING_CONFIG: dict[str, Any] = { - "max_tokens": 4000, - "compact_threshold": 0.5, - "target_usage": 0.3, - "protected_recent": 0.3, - "protected_tool_results": 2, - "truncate_chars": 40, - "compaction_notice_enabled": True, - "compaction_notice_min_level": 1, - "replay_last_user_on_compaction": True, -} - - -async def _add_tool_turn(context: SimpleContextManager, n: int) -> None: - await context.add_message({"role": "user", "content": f"user {n} " + "u" * 60}) - await context.add_message( - { - "role": "assistant", - "content": f"a {n}", - "tool_calls": [{"id": f"c{n}", "function": {"name": "t", "arguments": "{}"}}], - } - ) - await context.add_message( - {"role": "tool", "tool_call_id": f"c{n}", "content": "R" * 800} - ) - await context.add_message({"role": "assistant", "content": f"done {n} " + "d" * 60}) - - -# ---------------------------------------------------------------------- -# 1. Default off: byte-identical -# ---------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_default_is_off(): - """The flag defaults to False on both construction paths.""" - assert SimpleContextManager().replay_last_user_on_compaction is False - assert _make_context().replay_last_user_on_compaction is False - - -@pytest.mark.asyncio -async def test_default_off_is_byte_identical(): - """Two managers fed identical histories -- one constructed without the - flag at all, one with it explicitly False -- must produce byte-identical - views on every call, compaction included. - - Stronger than "the replay is absent": it asserts the whole returned - structure is unchanged, so a stray metadata key or a reordered tail - would fail too. - """ - baseline = _make_context() - explicit_off = _make_context(replay_last_user_on_compaction=False) - - for i in range(40): - for ctx in (baseline, explicit_off): - await ctx.add_message(_padded(i, "user")) - await ctx.add_message(_padded(i, "assistant")) - - got_baseline = await baseline.get_messages_for_request() - got_off = await explicit_off.get_messages_for_request() - assert _without_timestamps(got_baseline) == _without_timestamps(got_off) - - assert baseline._last_compaction_stats is not None, ( - "setup must actually trigger compaction for this test to mean anything" - ) - assert _replays(got_off) == [] - - -@pytest.mark.asyncio -async def test_default_off_never_touches_replay_state(): - """With the flag off, the boundary marker is never written -- proving - the guarded call site is the only thing that can reach this code.""" - context = _make_context() - await _fill_until_compacted(context) - await context.get_messages_for_request() - await context.get_messages_for_request() - assert context._last_replayed_boundary is None - - -# ---------------------------------------------------------------------- -# 2. Fires, and fires once per boundary -# ---------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_replay_appended_on_compaction(): - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - - view = await context.get_messages_for_request() - assert context._last_compaction_stats is not None - replays = _replays(view) - assert len(replays) == 1 - - replay = replays[0] - assert replay["role"] == "user" - assert replay["metadata"]["ephemeral"] is True - assert replay["content"].startswith( - f'' - ) - assert replay["content"].rstrip().endswith("") - assert "NOT a new request" in replay["content"] - - # It carries the actual most recent user text. - last_user = [ - m for m in view if m.get("role") == "user" and _is_real_user_message(m) - ][-1] - assert last_user["content"] in replay["content"] - - -@pytest.mark.asyncio -async def test_no_replay_without_compaction(): - """No compaction, no boundary, no replay.""" - context = _make_context(replay_last_user_on_compaction=True) - await context.add_message(_padded(0, "user")) - await context.add_message(_padded(0, "assistant")) - - view = await context.get_messages_for_request() - assert context._last_compaction_stats is None - assert _replays(view) == [] - assert context._last_replayed_boundary is None - - -@pytest.mark.asyncio -async def test_fires_once_per_boundary(): - """Repeated requests at the SAME compaction boundary emit exactly one - replay -- the first one. Subsequent calls, even as history keeps - growing and compaction keeps firing, must not re-emit until a new - boundary actually occurs. - - This is the test that distinguishes "once per boundary" from "once per - request", and it runs on a fixture whose sticky level is measured to - hold steady across the window it checks. - """ - context = SimpleContextManager(**ESCALATING_CONFIG) - - turn = 0 - emissions: list[int] = [] - levels: list[int] = [] - for _ in range(8): - await _add_tool_turn(context, turn) - turn += 1 - view = await context.get_messages_for_request() - emissions.append(len(_replays(view))) - levels.append(context._sticky_level) - - assert any(e == 1 for e in emissions), "the replay never fired at all" - - # Group the requests by the boundary in force at the time, and assert - # at most one emission per boundary. - per_boundary: dict[int, int] = {} - for level, emitted in zip(levels, emissions): - per_boundary[level] = per_boundary.get(level, 0) + emitted - for level, count in per_boundary.items(): - assert count <= 1, ( - f"replay fired {count} times within sticky level {level}: " - f"levels={levels} emissions={emissions}" - ) - - # And specifically: the level-3 boundary is held across several - # consecutive requests in this fixture, with exactly one emission. - assert levels.count(3) >= 2, f"fixture drifted; levels={levels}" - assert per_boundary[3] == 1 - - -@pytest.mark.asyncio -async def test_refires_on_a_new_boundary(): - """A genuinely new boundary (the sticky level escalating) re-arms the - replay -- each escalation re-buries the user's instruction deeper in - the view, which is the whole reason this feature exists.""" - context = SimpleContextManager(**ESCALATING_CONFIG) - - seen_levels: list[int] = [] - emissions: list[int] = [] - for turn in range(12): - await _add_tool_turn(context, turn) - view = await context.get_messages_for_request() - seen_levels.append(context._sticky_level) - emissions.append(len(_replays(view))) - - distinct_compacting_levels = {lv for lv in seen_levels if lv > 0} - assert len(distinct_compacting_levels) >= 2, ( - f"fixture must escalate at least once; levels={seen_levels}" - ) - assert sum(emissions) >= 2, ( - f"a new compaction boundary must re-arm the replay; " - f"levels={seen_levels} emissions={emissions}" - ) - - -@pytest.mark.asyncio -async def test_boundary_marker_resets_on_clear_and_set_messages(): - """A cleared or resumed session must not inherit a stale boundary - marker -- the boundary counters reset to (0, 0), so a stale marker - would silently suppress the first real replay of the new session.""" - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - await context.get_messages_for_request() - assert context._last_replayed_boundary is not None - - await context.clear() - assert context._last_replayed_boundary is None - - await _fill_until_compacted(context) - await context.get_messages_for_request() - assert context._last_replayed_boundary is not None - - await context.set_messages([_padded(0, "user")]) - assert context._last_replayed_boundary is None - - -# ---------------------------------------------------------------------- -# 3. Append-only: prefix / _seq / history invariants -# ---------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_prefix_stability_with_replay_enabled(): - """Enabling the replay must not disturb the byte-stable shared prefix. - - Both trailing ephemeral items (the replay and the compaction notice) - are stripped before comparison -- they are the dynamic tail by design. - The prefix in front of them must match byte for byte across a call - where history grew by exactly one turn. - """ - - def strip_trailing_ephemeral(messages: list[dict[str, Any]]) -> list[dict]: - out = list(messages) - while out and (out[-1].get("metadata") or {}).get("ephemeral"): - out.pop() - return out - - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - - call1 = strip_trailing_ephemeral(await context.get_messages_for_request()) - - await context.add_message(_padded(9001, "user")) - await context.add_message(_padded(9001, "assistant")) - - call2 = strip_trailing_ephemeral(await context.get_messages_for_request()) - - assert len(call2) >= len(call1) - assert call2[: len(call1)] == call1, ( - "enabling the last-user replay shifted the byte-stable prefix" - ) - - -@pytest.mark.asyncio -async def test_replay_does_not_mutate_history_or_consume_a_seq(): - """The replay is ephemeral: self.messages, `_seq` allocation, and every - sticky decision set must be exactly as they were.""" - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - - # Prime past the first compaction so state is settled. - await context.get_messages_for_request() - - history_before = copy.deepcopy(context.messages) - seq_before = context._next_seq - removed_before = set(context._removed_seqs) - truncated_before = set(context._truncated_seqs) - stubbed_before = set(context._stubbed_seqs) - - await context.set_messages(history_before) - await context.get_messages_for_request() - - assert context._next_seq == seq_before - assert len(context.messages) == len(history_before) - # A fresh set_messages resets sticky state, so compare shapes not sets. - assert isinstance(removed_before, set) - assert isinstance(truncated_before, set) - assert isinstance(stubbed_before, set) - - -@pytest.mark.asyncio -async def test_replay_carries_no_internal_seq_metadata(): - """Nothing internal leaks across the module boundary on the replay.""" - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - view = await context.get_messages_for_request() - - replays = _replays(view) - assert len(replays) == 1 - assert "_seq" not in (replays[0].get("metadata") or {}) - - -@pytest.mark.asyncio -async def test_replay_sits_before_the_compaction_notice(): - """The notice stays the final item -- the replay is the last item - BEFORE the dynamic tail, not after it.""" - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - view = await context.get_messages_for_request() - - sources = [(m.get("metadata") or {}).get("source") for m in view] - assert sources[-1] == "context-compaction" - assert sources[-2] == _REPLAY_ENVELOPE_SOURCE - - -# ---------------------------------------------------------------------- -# 4. No duplicate / nothing-to-say guards -# ---------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_no_duplicate_when_last_message_is_already_the_target(): - """When the view already ends with the most recent real user message, - a replay would be a byte-for-byte duplicate in the strongest position - the model already sees. Skip it.""" - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - # End the history on a user turn, so the compacted view's tail IS the - # most recent real user message. - await context.add_message(_padded(7777, "user")) - - view = await context.get_messages_for_request() - assert context._last_compaction_stats is not None - assert view[-1].get("role") == "user" - assert _replays(view) == [] - assert context._last_replayed_boundary is None, ( - "a skipped replay must leave the boundary unmarked so it can still " - "fire on a later request" - ) - - -@pytest.mark.asyncio -async def test_skips_when_tail_has_unanswered_tool_calls(): - """Tool-pair atomicity: never land a user-role message between an - assistant tool_call and its tool_result.""" - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - await context.add_message(_padded(8888, "assistant")) - await context.add_message( - { - "role": "assistant", - "content": "calling a tool", - "tool_calls": [ - {"id": "call_1", "function": {"name": "x", "arguments": "{}"}} - ], - } - ) - - view = await context.get_messages_for_request() - assert view[-1].get("tool_calls"), "setup must leave tool_calls at the tail" - assert _replays(view) == [] - assert context._last_replayed_boundary is None - - # Once the result arrives, the tail is safe again and the replay fires. - await context.add_message( - {"role": "tool", "tool_call_id": "call_1", "content": "done"} - ) - view2 = await context.get_messages_for_request() - assert len(_replays(view2)) == 1 - - -@pytest.mark.asyncio -async def test_skips_when_replay_would_blow_the_budget(): - """Compaction just shed tokens to reach the budget. A huge pasted user - message must not put the request straight back over it.""" - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - await context.add_message({"role": "user", "content": "H" * 200_000}) - await context.add_message(_padded(1, "assistant")) - - view = await context.get_messages_for_request() - assert context._last_compaction_stats is not None - assert _replays(view) == [], "replay overshot the budget it was given" - - -@pytest.mark.asyncio -async def test_never_replays_a_stub(): - """A stub is a placeholder that REPLACED the user's words, not the - words themselves -- replaying it under an envelope promising "a - verbatim copy of your most recent instruction" would make the envelope - lie. - - Exercised DIRECTLY rather than through a fixture, and honestly so: - this branch is unreachable through the normal path today, because the - ladder protects the last user message from stubbing at every level - (verified in _remove_messages_with_protection and Level 8). The guard - is defence in depth against that distant rule changing; a test that - always skips would prove nothing about it. - """ - context = _make_context(replay_last_user_on_compaction=True) - view = [ - {"role": "user", "content": '[User message compacted - original: "hi..."]', - "_stubbed": True, "_original_length": 999}, - {"role": "assistant", "content": "ack"}, - ] - context._maybe_append_last_user_replay(view, budget=10_000) - assert _replays(view) == [] - assert context._last_replayed_boundary is None - - -@pytest.mark.asyncio -async def test_skips_when_there_is_no_text_to_replay(): - """An image-only block list has no text; there is nothing to repeat.""" - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - await context.add_message( - {"role": "user", "content": [{"type": "image", "source": {"data": "..."}}]} - ) - await context.add_message(_padded(2, "assistant")) - - view = await context.get_messages_for_request() - assert _replays(view) == [] - - -@pytest.mark.asyncio -async def test_block_list_text_content_is_replayed(): - """Block-list content with text is replayed as its joined text.""" - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - await context.add_message( - { - "role": "user", - "content": [ - {"type": "text", "text": "first block"}, - {"type": "text", "text": "second block"}, - ], - } - ) - await context.add_message(_padded(3, "assistant")) - - view = await context.get_messages_for_request() - replays = _replays(view) - assert len(replays) == 1 - assert "first block\nsecond block" in replays[0]["content"] - - -# ---------------------------------------------------------------------- -# 5. The predicate: never replay a synthetic envelope -# ---------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_never_replays_a_reminder_envelope(): - """Given a history whose most recent user-role message is an ATTRIBUTED - reminder envelope (the shape this module's own summary message and this - feature's own replay both use), the replay must reach past it to the - real user turn -- or emit nothing. - - This is the regression test for the specific foundation gap documented - in `_is_real_user_message`: foundation 1.0.0 rejects only the BARE - `` tag, so an attributed envelope passes ITS check as - a real user turn. - """ - context = _make_context(replay_last_user_on_compaction=True) - await _fill_until_compacted(context) - await context.add_message( - { - "role": "user", - "content": '\nsynthetic\n', - } - ) - await context.add_message(_padded(4, "assistant")) - - view = await context.get_messages_for_request() - for replay in _replays(view): - assert "synthetic" not in replay["content"] - assert 'source="context-summary"' not in replay["content"] - - -def test_predicate_rejects_attributed_and_bare_envelopes(): - assert _is_real_user_message({"role": "user", "content": "hello"}) is True - assert ( - _is_real_user_message({"role": "user", "content": "x"}) - is False - ) - assert ( - _is_real_user_message( - {"role": "user", "content": 'x'} - ) - is False - ) - assert ( - _is_real_user_message( - { - "role": "user", - "content": [ - {"type": "text", "text": 'x'} - ], - } - ) - is False - ) - assert _is_real_user_message({"role": "assistant", "content": "hi"}) is False - assert ( - _is_real_user_message( - {"role": "user", "content": "r", "tool_call_id": "call_1"} - ) - is False - ) - - -# Frozen verbatim copy of amplifier_foundation.session.messages -# .is_real_user_message as of foundation 1.0.0 -- the version this module's -# vendored predicate was written against. Inlined rather than imported -# because foundation CANNOT be added even as a dev dependency here: -# foundation 1.0.0 requires amplifier-core>=1.0.10, and this repo pins -# amplifier-core from main at <1.0.10 (measured 2026-09-02, `uv sync` fails -# to resolve). A test that only ever skips is not a gate, so the reference -# is frozen here and the live-foundation check below is opportunistic on top. -def _foundation_1_0_0_reference(entry: dict[str, Any]) -> bool: - if entry.get("role") != "user": - return False - if "tool_call_id" in entry: - return False - content = entry.get("content", "") - if isinstance(content, str): - if content.strip().startswith(""): - return False - elif isinstance(content, list): - for block in content: - if isinstance(block, dict): - text = block.get("text", "") - if isinstance(text, str) and text.strip().startswith( - "" - ): - return False - return True - - -PREDICATE_CASES: list[dict[str, Any]] = [ - {"role": "user", "content": "plain"}, - {"role": "user", "content": " leading whitespace"}, - {"role": "user", "content": ""}, - {"role": "user", "content": "bare"}, - {"role": "user", "content": 'attr'}, - {"role": "user", "content": " indented"}, - {"role": "assistant", "content": "a"}, - {"role": "tool", "tool_call_id": "c", "content": "r"}, - {"role": "user", "content": "r", "tool_call_id": "c"}, - {"role": "user", "content": [{"type": "text", "text": "plain"}]}, - {"role": "user", "content": [{"type": "image", "source": {"data": "..."}}]}, - { - "role": "user", - "content": [{"type": "text", "text": "b"}], - }, -] - - -def test_predicate_is_strictly_stronger_than_foundation_1_0_0(): - """The relationship the vendored predicate claims, asserted against a - frozen copy of the foundation implementation it mirrors. - - Runs unconditionally -- no importorskip, no dependency. Everything - foundation rejects, this rejects too; the ONLY permitted divergence is - in the strengthening direction (this rejecting something foundation - accepts), and the specific case where that happens is pinned below so - the asymmetry stays documented rather than accidental. - """ - strengthened = [] - for case in PREDICATE_CASES: - theirs = _foundation_1_0_0_reference(case) - ours = _is_real_user_message(case) - if not theirs: - assert ours is False, ( - f"vendored predicate accepted what foundation rejects: {case!r}" - ) - elif not ours: - strengthened.append(case) - - assert strengthened == [ - { - "role": "user", - "content": 'attr', - } - ], ( - "the only intended divergence from foundation 1.0.0 is the " - f"ATTRIBUTED reminder envelope; got: {strengthened!r}" - ) - - -def test_live_foundation_still_has_the_attributed_envelope_gap(): - """Opportunistic drift alarm against the REAL foundation, wherever it - happens to be importable (it is not a dependency here -- see - _foundation_1_0_0_reference for why it cannot be one). - - If foundation ever closes the bare-tag gap, this fails loudly and the - hardening note in `_is_real_user_message` needs updating -- rather than - the frozen reference above quietly describing a foundation that no - longer exists. - """ - foundation = pytest.importorskip( - "amplifier_foundation.session.messages", - reason=( - "amplifier-foundation cannot be a dependency here (it requires " - "amplifier-core>=1.0.10; this repo pins core <1.0.10). The frozen " - "reference in test_predicate_is_strictly_stronger_than_foundation_1_0_0 " - "covers the same contract unconditionally." - ), - ) - - for case in PREDICATE_CASES: - assert foundation.is_real_user_message(case) == _foundation_1_0_0_reference( - case - ), f"frozen foundation reference has drifted from the real one: {case!r}" - - attributed = { - "role": "user", - "content": 'attr', - } - assert foundation.is_real_user_message(attributed) is True - assert _is_real_user_message(attributed) is False From 470c033c729ef734b7abef7fcefcc103a4b3cbf3 Mon Sep 17 00:00:00 2001 From: bkrabach <702425+bkrabach@users.noreply.github.com> Date: Wed, 2 Sep 2026 17:08:14 -0700 Subject: [PATCH 5/7] Revert "feat: tool-result budget (token-denominated, head+tail, per-tool) + spill-to-disk -- all defaults no-op (#21)" This reverts commit 49e27996b9b92bc37a7c29bd57d8b1d144ef1835. Merge policy: main carries wins only. Tool-result budget + spill-to-disk was an unproven, all-defaults-no-op feature. Belongs on a branch for evaluation, not on main. --- DONE-NOTE.md | 343 --------- README.md | 199 ----- amplifier_module_context_simple/__init__.py | 417 +---------- tests/test_tool_result_budget.py | 769 -------------------- 4 files changed, 2 insertions(+), 1726 deletions(-) delete mode 100644 tests/test_tool_result_budget.py diff --git a/DONE-NOTE.md b/DONE-NOTE.md index 4b822d7..ecc7ccc 100644 --- a/DONE-NOTE.md +++ b/DONE-NOTE.md @@ -1,19 +1,3 @@ - - -# Lane notes index - -| item | subject | -|---|---| -| `model_performance-x7p` | `protected_tool_results=0` protected ALL tool results (negative-slice bug) | -| `model_performance-x1r` | tool-result budget (token-denominated, head+tail, per-tool) + spill-to-disk | - ---- - # DONE-NOTE — model_performance-x7p `context-simple: protected_tool_results=0 protects ALL tool results (negative-slice bug)` @@ -318,330 +302,3 @@ Nothing blocking. Two observations handed on rather than acted on: is now well-defined at 0 and negative, but e.g. `target_usage=5.0` or `truncate_chars=-1` remain undefined. Worth one item if anyone wants the constructor to fail loud; not filed, because it is a design call, not a defect. - ---- - -# DONE-NOTE — W3-3 / `model_performance-x1r` - -**Fix the tool-result budget in context-simple** (250 chars head-only → token -budget, head+tail, per-tool), then spill the truncated middle to disk. - -| | | -|---|---| -| Repo | `amplifier-module-context-simple` | -| Branch | `lane/x1r-tool-result-budget` (forked from `c6dfbba`, `main`) | -| Draft PR | microsoft/amplifier-module-context-simple#21 (DRAFT — do not merge) | -| Tests | **151 green** (103 pre-existing, unchanged + 48 new) | -| Spend | **$0.00** — no API calls, no DTU, no containers, no infrastructure created | -| Default behavior | **byte-identical**, proven by external stash-compare (below) | - ---- - -## 1. Step 0 — the free measurement that decides how much this is worth - -The spec's own instruction: *"If our tool-result share is 15%, phase B is not -worth building and this spec should stop at phase A."* The magnitude case rested -on a practitioner's worked example from somebody else's workload (5 tool results -= 81% of a context). So it got measured on ours first. - -Script: `.amplifier/evaluation/treatment-validation/20260902-x1r/step0_tool_result_share.py` -Output: `.../20260902-x1r/step0-results.json` - -Source: every `transcript.jsonl` under two existing capture roots. Counted in -**characters** — the unit this module's estimator actually works in -(`len(str(msg)) // 4`). - -| | `20260902-t0t1` | `20260901-cadence` | -|---|---|---| -| sessions / tool results | 5 / 750 | 12 / 1,729 | -| **tool-result share of transcript chars** | **46.4%** | **47.3%** | -| p50 / p90 / p99 / max (chars) | 412 / 7,904 / 31,751 / 52,319 | 467 / 7,904 / 31,744 / 43,720 | -| over today's 250-char budget | 488 (65%) | 1,096 (63%) | -| chars discarded by today's budget | 1,524,013 (**91%** of tool-result content) | 3,082,300 (**90%**) | -| over 50,000 bytes | **1** | **0** | - -**(knob moved: none — observation) · (model family: n/a, corpus measurement) · -(confidence: measured, n=17 sessions / 2,479 tool results) · (evidence: -`.../20260902-x1r/step0-results.json`)** - -Two conclusions: - -1. **~47% ≫ 20%. Phase B is justified**, which is why this lane shipped spill and - not just the budget fix. -2. **deepseek's shipped 50,000-byte spill threshold would fire once in 2,479 - results on our workload.** The mass is in the 400–32,000 char band. The - mechanism is worth copying; that constant is not. This contradicts the - reference implementation the spill design otherwise follows, and it is the - single most transferable number in this note. - -Caveats, stated: a transcript is the **full history**, not the compacted wire -view, so this is the share of what compaction *sees* — an upper bound on the wire -share in a run that compacts. Token figures are the module's own estimator, not -provider-billed tokens. `p90 = 7,904` is identical in both roots, which suggests a -common fixed-size output rather than a coincidence; not investigated. - ---- - -## 2. What shipped - -Five flags, **all inert by default**: - -| flag | default | effect | -|---|---|---| -| `tool_result_budget_tokens` | `None` | Token-denominated budget. `None` = legacy `truncate_chars` path. | -| `tool_result_shape` | `"head"` | `"head_tail"` splits the budget and keeps both ends with `...[N chars omitted]...`. | -| `tool_result_budget_by_tool` | `{}` | Per-tool token budgets by tool name; beats the global budget. | -| `tool_result_exempt_tools` | `[]` | Never truncated (skill-type outputs). | -| `tool_result_spill_dir` | `None` | Full original written to a content-addressed file; pointer in the replacement text. | - -Precedence per result: **per-tool → global → legacy `truncate_chars`**. - -Roughly 300 LOC of module change against the spec's ~80 (phase A) + ~180 -(phase B) estimate — the overage is comment density and per-knob validation, not -extra mechanism. - -### Defaults rationale - -- **`tool_result_budget_tokens` defaults to `None`, not `62`.** The spec says 62 - (= today's 250 chars). But `truncate_chars` is an existing shipped knob: - someone running `truncate_chars: 500` today would have been silently reset to - 248 chars by a hard 62 default. `None` means "legacy path, whatever - `truncate_chars` says", which is byte-identical for **every** existing - configuration rather than only the default one. This is a deliberate deviation - from the spec, in the safer direction. -- **Recommended values are documented, not shipped.** The README's starter config - (2,000 tokens global, `head_tail`, per-tool map, `load_skill` exempt) is - conservative relative to codex's 10,000. It is *not* a default, because nothing - here has been evaluated against a model yet. -- **Per-tool numbers are transcribed, not measured.** They come from a - specification that publishes zero measurements anywhere, rescaled to the size - distribution in §1. The README says so at the point of use. -- **Tokens, not chars**, because chars/token constants are tokenizer-version - specific and drift (published up to 1.35×, observed up to 1.47× on technical - content) — a char budget silently changes meaning across a model version. The - conversion constant is `4`, deliberately the *same* constant the module's own - estimator uses, so budget and accounting cannot drift apart. -- **`head_tail` keeps the marker, not just the tail.** A model must never reason - from a truncated result without knowing it is truncated. - -### The one design constraint that shaped everything - -`_apply_sticky_decisions` re-derives the replacement text for **every** -sticky-truncated message on **every** request. So the replacement must be a pure -function of content + config. - -The consequence for spill: **the pointer is emitted whether or not the write -succeeded.** If the pointer tracked write success, one transient disk error would -change the bytes of an already-sent message, and under a grow-only prompt cache -every prefix mutation is a full cold rebuild. A dangling pointer is visible and -recoverable. A silently mutated prefix is neither. Write failures log a warning; -`tests/test_tool_result_budget.py::test_spill_write_failure_still_emits_stable_pointer` -pins it. - -Spill paths are content-addressed (`tool-result--.txt`), so -writes are idempotent across repeated requests and across a resumed session. -Write-then-rename, so a reader never sees a half-written file. - ---- - -## 3. Evidence - -### 3.1 Byte-identity — the defaults-are-a-no-op claim - -**Method (external oracle).** `.../20260902-x1r/byte_identity_harness.py` imports -whatever `amplifier_module_context_simple` is on `sys.path`, so neither side -defines its own baseline. 8 scenarios: light / heavy / aggressive pressure; -`truncate_chars` ∈ {10, 60, 250, 5000}; notice on and off; a view every third -turn **plus two consecutive views on turn 5** (the repeated call is what would -expose non-idempotent re-derivation). Canonical JSON of every returned message. -Only `metadata.timestamp` is normalized. - -``` -pre-change (c6dfbba, 2,648 lines, flags absent) sha256 76ee9d3f…3d6a467f -post-change (lane/x1r-tool-result-budget) sha256 76ee9d3f…3d6a467f -3,398,618 bytes IDENTICAL — PASS -``` - -**Non-vacuity:** the dump contains 76 `[truncated:` occurrences — the path is -really exercised. **Negative control:** the same harness with the treatment forced -on yields `4e59f380…` — a different hash, so the harness can see changes when -there are any. - -Artifacts: `byte-identity-PRE.json`, `byte-identity-POST.json`, -`byte-identity-TREATMENT.json`, `.log` files alongside. - -The unit suite pins the same claim from the opposite direction: -`test_default_config_replacement_text_is_byte_identical` asserts the exact legacy -string against an oracle **transcribed from the pre-change source**, not computed -by the new code, plus a second literal spelling so re-baselining the oracle alone -cannot silently pass. - -### 3.2 Tail retention — mechanism demonstration, no model, no spend - -`.../20260902-x1r/tail_retention_demo.py` → `tail-retention-demo.json`. -Four synthetic workloads (`pytest`, `grep`, `git log`, build output) whose answer -sits in the **last line** — the tool list enumerated *before* the run, as the gate -requires. - -| arm | truncated results | tail present | rate | -|---|---|---|---| -| control (shipped defaults, 250 chars, head) | 27 | 0 | **0%** | -| **budget-neutral** (62 tok = 248 chars, `head_tail`) | 28 | 28 | **100%** | -| 4× budget (250 tok = 1,000 chars, `head_tail`) | 1 | 1 | 100% | - -**(knob moved: `tool_result_shape`) · (model family: none — mechanical, no LLM) · -(confidence: measured, n=4 workloads × 30 turns per arm) · (evidence: -`.../20260902-x1r/tail-retention-demo.json`)** - -Row 2 is the clean A/B: **same bytes kept, different shape**, identical sticky -level per tool (4/2/3/3 in both arms). 0% → 100% tail retention at no budget cost. - -**Honest negative in row 3, and it is load-bearing.** Raising the per-result -budget *without* raising `target_usage`/`max_tokens` **trades truncation for -removal**: a bigger budget sheds fewer tokens per truncation, so the ladder -escalates past the truncation rungs into message removal — strictly more lossy. -Three of four workloads went from 11–15 truncated results to **zero, removed -instead**. Anyone tuning this must raise budget and target together, or measure -what they actually got. This is the first thing the follow-up eval should -control for. - -This is a mechanism demonstration, **not** the eval. No model was involved, so it -says nothing about whether an agent *uses* the tail. - -### 3.3 Test suite - -151 green. The 48 new tests cover: byte identity (literal + end-to-end + custom -`truncate_chars`); token budget arithmetic; head/tail split, exact omission -count, and the `content[-0:]`-returns-everything trap; per-tool resolution via -all three sources (harvested `tool_call_id`, `name` field, `metadata.tool_name`) -and both tool-call shapes (OpenAI `function`, Anthropic-ish); exemption including -"never counted as truncated"; spill write / idempotence / content-addressing / -**write-failure byte-stability** / lazy directory creation; **tool-pair atomicity -with every knob enabled**; `_seq` prefix stability and 5× request idempotence -with `head_tail` + spill on; `set_messages` map rebuild; `clear()` reset; config -validation fallbacks for all five flags; and `mount()` plumbing both ways. - -Twelve of them carry an explicit **"test must actually truncate/compact/spill"** -assertion, because a compaction test that escalates to level 8 removes every tool -result and then passes vacuously. That is exactly what the first draft of two of -these tests did. - ---- - -## 4. Discovered, filed, not fixed - -**`model_performance-x7p`** (filed `discovered-from` this item): -`protected_tool_results=0` protects **all** tool results, not none — -`tool_result_indices[-0:]` is the whole list (`__init__.py:1407`, `:1510`, -`:1577`). Compaction then skips every truncation rung and escalates straight to -removal. - -**(confidence: measured** — reproduced in a scratch harness: with -`protected_tool_results=0`, 40 turns × 2,000-char results reached sticky level 8 -with **0** truncations; `protected_tool_results=1` on the identical workload -truncated normally.**)** - -Not fixed here: the default is 5, so no shipped configuration hits it, and -changing it *is* a behavior change for anyone currently setting 0. ~3 lines plus a -regression test. The new tests avoid the trap and say why in a comment. - ---- - -## 5. Deviations from the spec, and why - -1. **Default is `None`, not `62`** — see §2. Protects existing `truncate_chars` - overrides. -2. **Spill lives in `context-simple`, not a new `tool-result-spill` module.** The - item's own task text asks for "spill-to-disk for the truncated middle with a - stable pointer line", which is a compaction-time operation on the truncated - middle — it shares the head/tail code and the `_seq` machinery. **This forfeits - the spec's class-A property**: because the content is already in the - conversation, spilling still shrinks the request, so it is still a boundary and - still a cold rebuild under a grow-only cache. It buys **recoverability**, not - free context. A post-execute hook that intercepts results *before* they enter - the conversation is the class-A design and remains unbuilt. Recorded plainly - rather than claimed. -3. **No cleanup sweep.** deepseek ships a 30-day startup sweep with symlink and - ownership guards. Nothing here deletes anything, ever — deliberately, since a - resumed or forked session may still hold an older pointer. The caller owns the - directory lifecycle; README says so and recommends a session-scoped dir. -4. **No `read`-result exemption by default.** deepseek exempts reads to prevent - `read → spill → read`. That loop cannot form here (spill happens at compaction - time, not tool-execute time). `tool_result_exempt_tools` is the seam if this - ever moves to a post-execute hook. -5. **No model-settable per-call budget.** codex lets the model raise its own limit - to 262,144 per call. That needs a tool-schema surface this module does not own. -6. **No line-based cap.** The spec's "chars before lines" concern is satisfied - structurally: the gate is a pure char count and no line cap exists in this - path, so the "2 lines × 10MB" failure mode is impossible rather than merely - unlikely. Zero LOC. -7. **No eval.** Explicitly out of scope for this lane ($0 authority). Spec below. - ---- - -## 6. Follow-up eval spec (the item this lane does NOT do) - -**Prerequisite that must be settled first:** the budget↔removal interaction in -§3.2. An arm that raises the budget while holding `target_usage` fixed is not -measuring "bigger budget", it is measuring "truncation replaced by removal". Fix -the design before spending. - -**Arms** (n≥3 each, one variable at a time, per the program's own rule): - -| arm | config | -|---|---| -| `control` | shipped defaults (byte-identical to today) | -| `shape_only` | `tool_result_budget_tokens: 62`, `tool_result_shape: "head_tail"` — **budget-neutral**, isolates shape | -| `budget_only` | `tool_result_budget_tokens: 2000`, shape `"head"`, `target_usage` raised to hold boundary count constant | -| `per_tool` | `budget_only` + `tool_result_budget_by_tool` + `tool_result_exempt_tools: ["load_skill"]` | -| `spill` | `per_tool` + `tool_result_spill_dir` | - -**Pre-register before spending** (a gate written after seeing the result is not a -gate; check each for vacuity): - -- **G-TRB-TAIL** — on a tool list enumerated **before** the run (pytest, grep, - git log, build output), the tail is present in the model-visible content of - 100% of truncated results; control 0%. *Already demonstrated mechanically - (§3.2); the eval's job is whether the agent* uses *it.* -- **G-TRB-RECALLS** — redundant re-calls of the same tool with identical - arguments fall vs control. **The real quality signal**, and the one that can - fail honestly. -- **G-TRB-COST** — cost does not rise. Watch the boundary count specifically: - §3.2 says a larger budget pushes work from truncation into removal, and prior - work established that boundary count, not the mechanism, is what moves cost - (a comparable feature measured +84% boundaries → +83% cost). -- **G-SPILL-APPEND** — no request contains a tool result a previous request - contained in longer form; `ID_ONLY = 0`. -- **G-SPILL-TOKENS** — tool-result input tokens fall by ≥ §1's predicted share, - ±20%. §1 is measured *before* the treatment, so it cannot be fitted afterwards. -- **G-SPILL-BOUNDARIES** — boundary count does not rise. -- **G-SPILL-RECOVERY** — **record as VACUOUS on S5-CRAC** and defer. S5 never - needs to recover anything (40/40 constraints, 20/20 post-compaction in every - run of every arm across six probes). It needs the discriminating scenario from - `model_performance-cb2`. -- **G-ANTH-GUARDRAIL** — required for any treatment lane: Anthropic re-warm ≤1 - request on raw wire fields, byte-stable system prompt, zero `ID_ONLY`. -- **S3 quality** — non-regression observation only. Six cells already sit at S3 - median 100 and dial-to-dial differences may be inside grader noise. - -**Cheapest first experiment:** `shape_only` vs `control`. One variable, no budget -change, therefore no boundary-count confound, and §3.2 already says the mechanism -fires. If G-TRB-RECALLS does not move there, it will not move anywhere. - ---- - -## 7. Infrastructure and spend - -**$0.00.** No API calls, no DTU, no Gitea, no containers. Nothing registered in -the infra ledger because nothing was created; nothing to tear down. All evidence -is local CPU work over existing capture roots. - -Artifacts written (outside the module repo, under the authorized capture path -`.amplifier/evaluation/treatment-validation/20260902-x1r/`): -`step0_tool_result_share.py`, `step0-results.json`, `byte_identity_harness.py`, -`byte-identity-{PRE,POST,TREATMENT}.json` + `.log`, `tail_retention_demo.py`, -`tail-retention-demo.json`. - -No PII, no team-internal data, no individual attribution in any output. Spill -paths are the only new content class in a transcript; they are caller-configured -and contain no absolute path this module invents. diff --git a/README.md b/README.md index 72ae0a8..c2d8055 100644 --- a/README.md +++ b/README.md @@ -439,205 +439,6 @@ absorptions cannot cycle; **hysteresis** on `summary_trigger` (arm at should be validated against the same cost metrics before this flag is enabled anywhere by default. -## Tool-result budget, shape, and spill - -**Every flag in this section defaults to a no-op.** With no configuration the -truncation path is byte-identical to what this module has always emitted: 250 -characters, head-only, the same prefix string. See "Byte-identity evidence" -below for how that is proven rather than asserted. - -### The defect this fixes - -When the progressive ladder truncates a tool result, it has always kept -`content[:250]` — **250 characters, head only, ~62 estimator tokens**. The one -shipped reference implementation available to compare against (codex) keeps a -**~10,000-token** budget, **head + tail**, with an explicit truncation marker, -and lets the model raise it per call up to 262,144. We were roughly **160× -below** it, on the *only* axis where a direct comparison exists. - -The head-only choice is the sharper half of the problem. For `pytest`, `grep`, -`git log` and build output **the answer is in the tail** — the failing -assertion, the matching line, the linker error. Keeping the head keeps the part -that matters least. - -### What the workload actually looks like (measured) - -From two existing capture roots — 17 sessions, 2,479 tool results, counted from -each session's own `transcript.jsonl` in **characters**, which is the unit this -module's estimator works in (`len(str(msg)) // 4`): - -| | `20260902-t0t1` | `20260901-cadence` | -|---|---|---| -| sessions / tool results | 5 / 750 | 12 / 1,729 | -| tool-result share of all transcript chars | **46.4%** | **47.3%** | -| size p50 / p90 / p99 / max (chars) | 412 / 7,904 / 31,751 / 52,319 | 467 / 7,904 / 31,744 / 43,720 | -| results over today's 250-char budget | 488 (65%) | 1,096 (63%) | -| chars discarded by today's budget | 1,524,013 (**91%** of all tool-result content) | 3,082,300 (**90%**) | -| results over 50,000 bytes | **1** | **0** | - -Two things follow, and the second one contradicts the reference -implementation this spill design otherwise copies: - -1. Tool results are **~47% of the material compaction has to work with**, not a - rounding error. (Caveat, stated: a transcript is the full history, not the - compacted wire view, so this is the share of what compaction *sees* — an - upper bound on the wire share in a run that compacts. Confidence: measured, - char-denominated; token figures are the module's own estimator, not - provider-billed tokens.) -2. deepseek's shipped **50,000-byte spill threshold would fire once in 2,479 - results** on this workload. The mass sits in the 400–32,000 char band. Do not - copy that constant; copy the mechanism. - -### The flags - -| flag | default | effect | -|---|---|---| -| `tool_result_budget_tokens` | `None` | Token-denominated budget. `None` keeps the legacy `truncate_chars` char budget, byte-identical. | -| `tool_result_shape` | `"head"` | `"head_tail"` splits the budget in half and keeps both ends with an explicit `...[N chars omitted]...` marker. | -| `tool_result_budget_by_tool` | `{}` | Per-tool token budgets keyed by tool name. Takes precedence over the global budget. | -| `tool_result_exempt_tools` | `[]` | Tool results that are **never** truncated. | -| `tool_result_spill_dir` | `None` | Writes the full original result to a content-addressed file and points the model at it. `None` writes nothing, ever. | - -Precedence for a single result: **per-tool budget → global token budget → -legacy `truncate_chars`**. Setting only `tool_result_budget_by_tool` therefore -leaves every *other* tool on the legacy path, byte-identical. - -A starter configuration — conservative relative to codex's 10,000, and -deliberately **not** the shipped default: - -```toml -[[contexts]] -module = "context-simple" -config = { - tool_result_budget_tokens = 2000, # ~8,000 chars, vs today's 250 - tool_result_shape = "head_tail", - tool_result_budget_by_tool = { grep = 1000, read_file = 4000, bash = 2000 }, - tool_result_exempt_tools = ["load_skill"], # never trim skill output - tool_result_spill_dir = "/tmp/amplifier-spill", -} -``` - -These per-tool numbers are **starting points transcribed from a specification -that publishes no measurements**, rescaled to the size distribution above. They -are not measured. Say so wherever they get copied. - -### Why tokens, not chars - -Chars-per-token constants are tokenizer-version specific and drift (published -drift up to 1.35×, observed up to 1.47× on technical content, with the rate card -unchanged). A char budget therefore silently changes meaning across a model -version; a token budget does not. The conversion constant used here is `4` — -deliberately the same constant this module's own estimator uses, so a budget -expressed in tokens and the accounting the ladder runs on cannot drift apart. - -**Chars before lines, always.** The gate is a pure character count and there is -no line-based cap anywhere in this path, so the "a file could have 2 lines that -are each 10MB" failure mode is structurally impossible here rather than merely -unlikely. - -### Spill: the truncated middle stays reachable - -With `tool_result_spill_dir` set, the **full original** result is written to -`/tool-result--.txt` and the replacement text becomes: - -``` -[truncated: ~12,431 tokens - read /tmp/amplifier-spill/tool-result-000042-a1b2….txt for the full result] -...[48,920 chars omitted]... - -``` - -There is **no new retrieval tool**, deliberately: the model is pointed at the -ordinary file tools it already has. (The opposite failure is on record — a -README advertising `grep` over an archive where only `ls`/`view` existed, leaving -the model to guess which summary hid the fact it needed.) - -Three properties worth knowing: - -- **The pointer is a pure function of content + config, and is emitted whether - or not the write succeeded.** This is not sloppiness, it is the constraint: - `_apply_sticky_decisions` re-derives the replacement text for every - sticky-truncated message on *every request*. A pointer that tracked write - success would change the bytes of an already-sent message after a transient - disk error — and under a grow-only prompt cache, every prefix mutation is a - **full cold rebuild**. A dangling pointer is visible and recoverable; a - silently mutated prefix is neither. Write failures log a warning. -- **Content-addressed, so writes are idempotent** across repeated requests and - across a resumed or forked session. -- **Nothing is ever deleted** — not on `clear()`, not on compaction, not at - session end. A resumed or forked session may still hold a pointer to an older - file. **No cleanup sweep ships in this module**: the caller owns the - directory's lifecycle, and a session-scoped directory is the recommended - shape. (deepseek's reference implementation ships a 30-day startup sweep with - symlink and ownership guards; porting it is deliberately left to whoever wires - spill into a bundle default.) - -### Shape change is nearly free; budget change is not - -Mechanism demonstration, no model and no spend — 4 synthetic tool workloads -(`pytest`, `grep`, `git log`, build output) whose answer sits in the last line, -enumerated *before* the run: - -| arm | truncated results | tail present | rate | -|---|---|---|---| -| control (shipped defaults: 250 chars, head) | 27 | 0 | **0%** | -| **budget-neutral** (62 tokens = 248 chars, `head_tail`) | 28 | 28 | **100%** | -| 4× budget (250 tokens = 1,000 chars, `head_tail`) | 1 | 1 | 100% | - -The middle row is the clean A/B: **same bytes kept, different shape**, same -sticky level per tool (4/2/3/3 in both arms). Tail retention goes 0% → 100% at -no budget cost. - -The third row is the honest caveat, and it is load-bearing for anyone tuning -this: **raising the per-result budget without also raising `target_usage` or -`max_tokens` trades truncation for removal.** A larger budget sheds fewer tokens -per truncation, so the ladder escalates past the truncation rungs into message -*removal* — which is strictly more lossy. Three of four workloads went from -11–15 truncated results to **zero**, having been removed instead. Raise the -budget and the target together, or measure what you actually got. - -### Byte-identity evidence - -Claim: with default configuration, output is byte-identical to before this -change. Method: an **external** harness (it imports whatever -`amplifier_module_context_simple` is on `sys.path`, so neither side defines its -own baseline) drives 8 scenarios — light/heavy/aggressive pressure, four -`truncate_chars` values from 10 to 5,000, notice on and off — takes a view every -third turn plus two consecutive views on turn 5, and dumps canonical JSON of -every returned message. Only `metadata.timestamp` is normalized; every truncated -result is compared character for character. - -``` -pre-change (c6dfbba, 2,648 lines, no such flags) sha256 76ee9d3f…a467f -post-change (this branch) sha256 76ee9d3f…a467f -3,398,618 bytes, identical PASS -``` - -Non-vacuity: the dump contains 76 `[truncated:` occurrences, so the harness -really exercises the path. Negative control: the same harness with the treatment -forced on produces a **different** hash, so it can see changes when there are -any. - -The unit suite pins the same claim from the other direction — -`tests/test_tool_result_budget.py` asserts the exact legacy replacement string -literally, against an oracle transcribed from the pre-change source rather than -computed by the new code. - -### Not built here - -- **Model-settable per-call budgets.** codex lets the model raise its own - truncation limit per call up to 262,144 tokens. That needs a tool-schema - surface this module does not own. -- **Count-based spill triggers inside the search tools** (`glob` at >100 - results, `grep` at >250 matches). Size is not the only signal, but those - triggers belong in the tools, not in the context manager. -- **`read`-result exemption.** deepseek exempts read results specifically to - prevent a `read → spill → read again` loop. Here spill happens *at compaction - time* on content already in the conversation, not at tool-execute time before - it enters, so that loop cannot form — but if this ever moves to a - post-execute hook, the exemption becomes necessary. `tool_result_exempt_tools` - is the seam for it. -- **A cleanup sweep.** See above. - ## Dependencies - `amplifier-core>=1.0.0` diff --git a/amplifier_module_context_simple/__init__.py b/amplifier_module_context_simple/__init__.py index 77ad0c7..bde6d17 100644 --- a/amplifier_module_context_simple/__init__.py +++ b/amplifier_module_context_simple/__init__.py @@ -94,61 +94,12 @@ here; see README for the candidate levers (cooldown / absolute floor / trigger hysteresis). OPT-IN, EXPERIMENTAL -- do not enable by default. - -Tool-result budget and spill (opt-in, default off -- see config -`tool_result_budget_tokens` / `tool_result_shape` / -`tool_result_budget_by_tool` / `tool_result_exempt_tools` / -`tool_result_spill_dir`): - • The truncation rung of the progressive ladder historically kept - `content[:250]` -- 250 characters, HEAD ONLY, ~62 estimator tokens. - The only shipped reference implementation available to compare - against (codex) keeps a ~10,000-token budget, HEAD + TAIL, with an - explicit truncation marker. We were ~160x below it, and keeping the - head is the part that hurts: for `pytest`, `grep`, `git log` and - build output the ANSWER IS IN THE TAIL. - • MEASURED (step 0 of this change, two existing capture roots, 17 - sessions, 2,479 tool results, char-denominated, confidence: - measured): tool-result content is 46.4% / 47.3% of all transcript - characters; individual results run p50 412 / p90 7,904 / p99 ~31.7k - / max 52.3k chars; 63-65% of every tool result exceeds today's - 250-char budget, and today's budget discards ~91% of all - tool-result content it touches. - • `tool_result_budget_tokens: int` replaces the char budget with a - TOKEN-denominated one (chars/token constants are tokenizer-version - specific and drift; a char budget silently changes meaning across a - model version). Default `None` = the pre-existing `truncate_chars` - path, byte-identical. - • `tool_result_shape: "head" | "head_tail"` -- `"head_tail"` splits - the budget in half and keeps both ends with an explicit - `...[N chars omitted]...` marker between them, so the model can never - reason from a truncated result without knowing it was truncated. - Default `"head"`. - • `tool_result_budget_by_tool: dict[str, int]` -- per-tool token - budgets, resolved by tool name; takes precedence over the global - budget. Default `{}`. - • `tool_result_exempt_tools: list[str]` -- these tool results are - NEVER truncated (the value-order names loaded skills explicitly). - Default `[]`. - • `tool_result_spill_dir: path` -- when set, the FULL original result - is written to a content-addressed file under this directory and the - replacement text points the model at it, so the truncated middle is - recoverable through the ordinary file tools with no new retrieval - tool. Default `None` (nothing is ever written). - • DETERMINISM, load-bearing: the replacement text (including the spill - pointer) is a pure function of the message's content plus config. It - NEVER depends on whether the spill write succeeded, because - `_apply_sticky_decisions` re-derives it on every single request and a - pointer that changed after a failed-then-successful write would - mutate an already-cached prefix -- which, under a grow-only prompt - cache, is a full cold rebuild. A failed write yields a pointer to a - missing file (visible, recoverable) rather than a silent byte change. """ # Amplifier module metadata __amplifier_module_type__ = "context" import asyncio -import hashlib import json import logging from collections.abc import Awaitable, Callable @@ -197,23 +148,6 @@ COMPACTION_STRATEGY_SUMMARY, ) -# tool_result_shape config values. "head" (default) keeps the leading slice -# only -- the pre-existing behavior. "head_tail" splits the budget in half -# and keeps both ends with an explicit omission marker between them. See -# module docstring "Tool-result budget and spill". -TOOL_RESULT_SHAPE_HEAD = "head" -TOOL_RESULT_SHAPE_HEAD_TAIL = "head_tail" -_VALID_TOOL_RESULT_SHAPES = (TOOL_RESULT_SHAPE_HEAD, TOOL_RESULT_SHAPE_HEAD_TAIL) - -# Chars-per-token constant used to convert a token-denominated tool-result -# budget into the char slice actually taken. Deliberately the SAME constant -# this module's own estimator uses (_estimate_tokens: len(str(msg)) // 4), so -# a budget expressed in tokens and the token accounting the compaction ladder -# runs on cannot drift apart. This is an estimator, not a tokenizer: see -# README "Tool-result budget" for why the budget is nevertheless expressed in -# tokens rather than chars. -_TOOL_RESULT_CHARS_PER_TOKEN = 4 - # The summary message's envelope source tag and metadata type marker. The # envelope is what makes foundation's is_real_user_message() classify this # role="user" message as NOT a real user turn (see module docstring); the @@ -311,26 +245,6 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = provider.complete() call before treating it as a failure and falling back to progressive compaction for that pass (default: 30.0). - - tool_result_budget_tokens: Token budget kept when truncating a - tool result (default: None = use the legacy `truncate_chars` - char budget, byte-identical to before this feature existed). - Setting it switches that tool result to the token-denominated - path. See module docstring "Tool-result budget and spill". - - tool_result_shape: "head" (default) or "head_tail". "head_tail" - splits the budget in half and keeps both ends with an explicit - `...[N chars omitted]...` marker. An unrecognized value falls - back to "head" with a logged warning rather than crashing - mount(). - - tool_result_budget_by_tool: Per-tool token budgets keyed by tool - name (default: {}). Takes precedence over - tool_result_budget_tokens for a matching tool. Entries whose - value is not a positive int are dropped with a logged warning. - - tool_result_exempt_tools: Tool names whose results are NEVER - truncated (default: []). Recommended for skill-type outputs. - - tool_result_spill_dir: Directory to write the FULL original tool - result into when it is truncated, so the truncated middle stays - recoverable via the ordinary file tools (default: None = nothing - is ever written and no pointer appears in any message). Returns: Cleanup callable that unregisters the token-meter hook (if one was @@ -378,11 +292,6 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = summarization_model=config.get("summarization_model"), summarization_prompt_path=config.get("summarization_prompt_path"), summarization_timeout_s=config.get("summarization_timeout_s", 30.0), - tool_result_budget_tokens=config.get("tool_result_budget_tokens"), - tool_result_shape=config.get("tool_result_shape", TOOL_RESULT_SHAPE_HEAD), - tool_result_budget_by_tool=config.get("tool_result_budget_by_tool"), - tool_result_exempt_tools=config.get("tool_result_exempt_tools"), - tool_result_spill_dir=config.get("tool_result_spill_dir"), hooks=getattr(coordinator, "hooks", None), ) @@ -464,11 +373,6 @@ def __init__( summarization_model: str | None = None, summarization_prompt_path: str | None = None, summarization_timeout_s: float = 30.0, - tool_result_budget_tokens: int | None = None, - tool_result_shape: str = TOOL_RESULT_SHAPE_HEAD, - tool_result_budget_by_tool: dict[str, int] | None = None, - tool_result_exempt_tools: list[str] | None = None, - tool_result_spill_dir: str | None = None, hooks: Any = None, ): """ @@ -517,18 +421,6 @@ def __init__( DEFAULT_SUMMARIZATION_PROMPT. None uses the built-in prompt. summarization_timeout_s: Seconds to wait for the summarizer's provider.complete() call before treating it as a failure. - tool_result_budget_tokens: Token budget kept when truncating a - tool result. None (default) keeps the pre-existing - `truncate_chars` char budget, byte-identical. - tool_result_shape: "head" (default, pre-existing behavior) or - "head_tail". An unrecognized value falls back to "head" with - a logged warning rather than raising. - tool_result_budget_by_tool: Per-tool token budgets keyed by tool - name; takes precedence over tool_result_budget_tokens. - tool_result_exempt_tools: Tool names whose results are never - truncated. - tool_result_spill_dir: Directory the full original result is - written to when truncated. None (default) writes nothing. hooks: Optional hooks instance for emitting observability events and (always, when present) recording real usage for the token meter via `llm:response` -- see `_on_llm_response`. @@ -565,42 +457,6 @@ def __init__( self.summarization_model = summarization_model self.summarization_prompt_path = summarization_prompt_path self.summarization_timeout_s = summarization_timeout_s - # --- Tool-result budget / shape / spill (all default no-op) --- - # Every one of these is validated the same way the other enums in - # this module are: an unusable value logs a warning and falls back to - # the pre-existing behavior rather than crashing mount(). A context - # manager that refuses to start is worse than one that runs at the - # old default and says so. - self.tool_result_budget_tokens = self._validate_budget_tokens( - tool_result_budget_tokens - ) - if tool_result_shape not in _VALID_TOOL_RESULT_SHAPES: - logger.warning( - f"context-simple: unknown tool_result_shape {tool_result_shape!r} " - f"(expected one of {_VALID_TOOL_RESULT_SHAPES!r}); falling back to " - f"{TOOL_RESULT_SHAPE_HEAD!r}" - ) - tool_result_shape = TOOL_RESULT_SHAPE_HEAD - self.tool_result_shape = tool_result_shape - self.tool_result_budget_by_tool = self._validate_budget_by_tool( - tool_result_budget_by_tool - ) - self.tool_result_exempt_tools = frozenset( - str(t) for t in (tool_result_exempt_tools or []) if str(t) - ) - self.tool_result_spill_dir = ( - str(tool_result_spill_dir) if tool_result_spill_dir else None - ) - # tool_call_id -> tool name, harvested from assistant tool_calls so a - # tool RESULT (which carries only the id) can be matched to a per-tool - # budget or exemption. Only populated when at least one of those two - # knobs is configured -- in the default configuration this map stays - # empty and add_message() does no extra work at all. - self._tool_name_by_call_id: dict[str, str] = {} - # Spill paths already written this session, so the same result is not - # re-written on every request (_apply_sticky_decisions re-derives the - # replacement text for every sticky-truncated message, every call). - self._spilled_paths: set[str] = set() self._hooks = hooks self._last_compaction_stats: dict[str, Any] | None = None # --- Summary compaction strategy state (compaction_strategy == "summary") --- @@ -701,12 +557,6 @@ async def add_message(self, message: dict[str, Any]) -> None: } self._next_seq += 1 - # Harvest tool_call_id -> tool name for per-tool budgets/exemptions. - # Gated on config: in the default configuration this is a single - # boolean check and nothing else. - if self._per_tool_config_active(): - self._harvest_tool_names(message) - # Add message (no rejection - compaction happens ephemerally) self.messages.append(message) @@ -1034,17 +884,10 @@ async def set_messages(self, messages: list[dict[str, Any]]) -> None: produced the transcript. """ restamped: list[dict[str, Any]] = [] - per_tool_active = self._per_tool_config_active() - self._tool_name_by_call_id = {} for i, msg in enumerate(messages): meta = dict(msg.get("metadata") or {}) meta["_seq"] = i restamped.append({**msg, "metadata": meta}) - # Rebuild the id->name map from the restored history, so a - # resumed session resolves per-tool budgets exactly as the - # original session did. No-op unless a per-tool knob is set. - if per_tool_active: - self._harvest_tool_names(msg) self.messages = restamped self._next_seq = len(restamped) # Seqs were just restamped from 0, so any anchor split recorded @@ -1072,12 +915,6 @@ async def clear(self) -> None: self._last_measured_prompt_tokens = None self._last_token_meter_stats = None self._reset_hybrid_meter_state() - # Per-tool name map belongs to the cleared conversation. The spill - # cache is only a write-skip memo -- the files themselves are - # content-addressed and deliberately NOT deleted here (a resumed or - # forked session may still hold a pointer to one; see README). - self._tool_name_by_call_id = {} - self._spilled_paths = set() self._reset_summary_strategy_state() logger.info("Context cleared") @@ -2157,13 +1994,6 @@ def _truncate_tool_wave( msg = messages[i] if msg.get("role") != "tool": # Verify it's still a tool message continue - if self._tool_result_is_exempt(msg): - # Skipped BEFORE _record_truncated so an exempt result is - # never marked truncated (which would inflate the reported - # truncation count and pin a permanent sticky decision that - # does nothing). No-op unless tool_result_exempt_tools is - # configured. - continue if not msg.get("_truncated"): if true_total is None: # First mutation in this call: establish the true baseline @@ -2573,263 +2403,20 @@ async def _finalize_compaction_with_stats( return final_messages - # --- Tool-result budget / shape / spill helpers --- - # - # Every one of these is a no-op in the default configuration. See the - # module docstring "Tool-result budget and spill" for the measured - # rationale and the determinism constraint that shapes the spill design. - - @staticmethod - def _validate_budget_tokens(value: Any) -> int | None: - """Coerce `tool_result_budget_tokens` to a positive int, or None. - - None (the default) means "keep the pre-existing char budget". - Anything unusable logs a warning and becomes None -- i.e. falls back - to today's behavior rather than raising at mount time. - """ - if value is None: - return None - if isinstance(value, bool) or not isinstance(value, int) or value < 1: - logger.warning( - f"context-simple: tool_result_budget_tokens must be a positive " - f"int, got {value!r}; falling back to the truncate_chars " - f"char budget" - ) - return None - return value - - @staticmethod - def _validate_budget_by_tool(value: Any) -> dict[str, int]: - """Coerce `tool_result_budget_by_tool` to {tool_name: positive int}. - - Unusable entries are dropped INDIVIDUALLY with a warning naming the - offending key, so one bad entry never silently discards the whole map - (and never crashes mount()). - """ - if not value: - return {} - if not isinstance(value, dict): - logger.warning( - f"context-simple: tool_result_budget_by_tool must be a dict, " - f"got {type(value).__name__}; ignoring it" - ) - return {} - cleaned: dict[str, int] = {} - for name, budget in value.items(): - if ( - isinstance(budget, bool) - or not isinstance(budget, int) - or budget < 1 - or not str(name) - ): - logger.warning( - f"context-simple: tool_result_budget_by_tool[{name!r}] must " - f"be a positive int, got {budget!r}; dropping this entry" - ) - continue - cleaned[str(name)] = budget - return cleaned - - def _per_tool_config_active(self) -> bool: - """True when any knob needs a tool RESULT matched to a tool NAME.""" - return bool(self.tool_result_budget_by_tool or self.tool_result_exempt_tools) - - def _harvest_tool_names(self, message: dict[str, Any]) -> None: - """Record tool_call_id -> tool name from an assistant message. - - A tool result carries only `tool_call_id`; the NAME lives on the - assistant message that requested the call. This harvests the mapping - incrementally as history is appended, so per-tool budget lookup is - O(1) and never rescans history. Handles both the OpenAI - (`{"id", "function": {"name"}}`) and Anthropic-ish - (`{"id", "name"}`) tool_call shapes -- the same two shapes - _format_messages_for_summarization already handles. - - Called only when a per-tool knob is configured: in the default - configuration this never runs. - """ - for tc in message.get("tool_calls") or []: - if not isinstance(tc, dict): - continue - call_id = tc.get("id") or tc.get("tool_call_id") - if "function" in tc and isinstance(tc["function"], dict): - name = tc["function"].get("name") - else: - name = tc.get("name") or tc.get("tool") - if call_id and name: - self._tool_name_by_call_id[str(call_id)] = str(name) - - def _resolve_tool_name(self, msg: dict[str, Any]) -> str | None: - """Best-effort tool name for a tool RESULT message. - - Three sources, in order of directness. Returning None simply means - no per-tool rule can apply to this message -- it falls through to the - global budget, which is the safe direction. - """ - name = msg.get("name") - if isinstance(name, str) and name: - return name - meta_name = (msg.get("metadata") or {}).get("tool_name") - if isinstance(meta_name, str) and meta_name: - return meta_name - call_id = msg.get("tool_call_id") - if call_id: - return self._tool_name_by_call_id.get(str(call_id)) - return None - - def _tool_result_is_exempt(self, msg: dict[str, Any]) -> bool: - """True if this tool result must never be truncated.""" - if not self.tool_result_exempt_tools: - return False - name = self._resolve_tool_name(msg) - return name is not None and name in self.tool_result_exempt_tools - - def _resolve_tool_result_budget(self, msg: dict[str, Any]) -> tuple[int, str]: - """Return (budget_chars, shape) for one tool result. - - Precedence: per-tool budget > global token budget > the pre-existing - `truncate_chars` char budget. The shape knob applies to whichever - budget won -- there is deliberately no per-tool shape, because a - per-tool budget with a global shape already covers every case the - reference implementations express, and a second per-tool map would - double the config surface for no measured gain. - """ - budget_tokens: int | None = None - if self.tool_result_budget_by_tool: - name = self._resolve_tool_name(msg) - if name is not None: - budget_tokens = self.tool_result_budget_by_tool.get(name) - if budget_tokens is None: - budget_tokens = self.tool_result_budget_tokens - if budget_tokens is None: - # Nothing configured: the pre-existing char budget, head-only. - # This is the byte-identical default path. - return self.truncate_chars, TOOL_RESULT_SHAPE_HEAD - return budget_tokens * _TOOL_RESULT_CHARS_PER_TOKEN, self.tool_result_shape - - def _spill_pointer(self, msg: dict[str, Any], content: str) -> str | None: - """Write the full result to the spill dir; return its path, or None. - - DETERMINISM (load-bearing -- see module docstring): the returned path - is a pure function of `content` (+ the message's stable `_seq`) and - is returned REGARDLESS of whether the write succeeded. This method is - re-entered for every sticky-truncated message on every request; if - the pointer text tracked write success, one transient disk error - would change the bytes of an already-sent message and cold-rebuild - the prompt cache. A dangling pointer is visible and recoverable; a - silently mutated prefix is neither. - - Content-addressing also makes the write idempotent across a resumed - or forked session: the same result always lands at the same path. - """ - if not self.tool_result_spill_dir: - return None - digest = hashlib.sha256(content.encode("utf-8", errors="replace")).hexdigest()[ - :16 - ] - seq = self._extract_seq(msg) - stem = ( - f"tool-result-{seq:06d}-{digest}" - if isinstance(seq, int) - else f"tool-result-{digest}" - ) - path = Path(self.tool_result_spill_dir) / f"{stem}.txt" - path_str = str(path) - if path_str in self._spilled_paths: - return path_str - try: - path.parent.mkdir(parents=True, exist_ok=True) - if not path.exists(): - # Write-then-rename so a reader (the agent, via its file - # tools) can never observe a half-written spill file. - tmp = path.parent / f"{path.name}.{id(self):x}.tmp" - tmp.write_text(content, encoding="utf-8") - tmp.replace(path) - self._spilled_paths.add(path_str) - except Exception as e: - # Deliberately NOT fatal and deliberately NOT reflected in the - # returned pointer -- see the determinism note above. - logger.warning( - f"context-simple: could not spill tool result to {path_str}: {e}. " - f"The pointer is still emitted (byte-stability); the file may " - f"be missing." - ) - return path_str - - def _format_truncated_tool_result( - self, - content: str, - budget_chars: int, - shape: str, - original_tokens: int, - spill_path: str | None, - ) -> str: - """Build the replacement text for one over-budget tool result. - - BYTE-IDENTITY CONTRACT: with `budget_chars == self.truncate_chars`, - `shape == "head"` and `spill_path is None` -- i.e. the default - configuration -- this returns EXACTLY the string this module has - always returned. tests/test_tool_result_budget.py pins that literally. - """ - recovery = ( - f"read {spill_path} for the full result" - if spill_path - else "call tool again if needed" - ) - header = f"[truncated: ~{original_tokens:,} tokens - {recovery}]" - - if shape != TOOL_RESULT_SHAPE_HEAD_TAIL: - return f"{header} {content[:budget_chars]}..." - - # head_tail: split the budget in half, keep both ends, and say - # explicitly how much vanished in between. The marker matters as much - # as the tail does -- a model must never reason from a truncated - # result without knowing that it is truncated. - head_chars = budget_chars // 2 - tail_chars = budget_chars - head_chars - # content[-0:] is the WHOLE string, not the empty one; guard it. - tail = content[-tail_chars:] if tail_chars else "" - omitted = len(content) - head_chars - tail_chars - return ( - f"{header} {content[:head_chars]}" - f"\n...[{omitted:,} chars omitted]...\n" - f"{tail}" - ) - def _truncate_tool_result(self, msg: dict[str, Any]) -> dict[str, Any]: """ Truncate a tool result message to reduce token count. Returns a NEW dict - does not modify the original. - - Deterministic: called once per over-budget tool result during an - escalation, and again for every sticky-truncated message on every - subsequent request (_apply_sticky_decisions). Same input, same bytes, - every time -- that is what the returned view's prefix stability rests - on. """ content = msg.get("content", "") - if not isinstance(content, str): - return msg - if self._tool_result_is_exempt(msg): - # No-op unless tool_result_exempt_tools is configured. - return msg - - budget_chars, shape = self._resolve_tool_result_budget(msg) - # CHARS BEFORE LINES: the gate below is, and has always been, a pure - # character count. There is no line-based cap anywhere in this path, - # so the "a file could have 2 lines that are each 10MB" failure mode - # is structurally impossible here rather than merely unlikely. - if len(content) <= budget_chars: + if not isinstance(content, str) or len(content) <= self.truncate_chars: return msg original_tokens = len(content) // 4 - spill_path = self._spill_pointer(msg, content) return { **msg, - "content": self._format_truncated_tool_result( - content, budget_chars, shape, original_tokens, spill_path - ), + "content": f"[truncated: ~{original_tokens:,} tokens - call tool again if needed] {content[: self.truncate_chars]}...", "_truncated": True, "_original_tokens": original_tokens, } diff --git a/tests/test_tool_result_budget.py b/tests/test_tool_result_budget.py deleted file mode 100644 index 885dcc7..0000000 --- a/tests/test_tool_result_budget.py +++ /dev/null @@ -1,769 +0,0 @@ -"""Tests for the tool-result budget, shape, per-tool rules, and spill. - -Everything this feature adds is OFF by default. The single most important -test in this file is `test_default_config_replacement_text_is_byte_identical`: -it pins the exact replacement string this module has emitted since before the -feature existed, character for character. If that test ever fails, the -"defaults are a no-op" claim in the README and the module docstring is false. - -The other three load-bearing groups: - - - TOOL-PAIR INTEGRITY: turning any of these knobs on must not break the - tool_use/tool_result atomicity the provider APIs require. A donor engine - measured 29 of 30 turns dying on exactly this. - - - _seq / PREFIX STABILITY: the replacement text (including the spill - pointer) is re-derived for every sticky-truncated message on every single - request. It must be byte-identical every time, or the shared prefix - mutates and -- under a grow-only prompt cache -- every mutation is a full - cold rebuild. - - - SPILL WRITE FAILURE: a failed write must NOT change the emitted bytes. - See `test_spill_write_failure_still_emits_stable_pointer`. -""" - -import logging -from pathlib import Path -from typing import Any - -import pytest -from amplifier_module_context_simple import ( - TOOL_RESULT_SHAPE_HEAD, - TOOL_RESULT_SHAPE_HEAD_TAIL, - SimpleContextManager, -) - -# -------------------------------------------------------------------------- -# helpers -# -------------------------------------------------------------------------- - - -def _make_context(**overrides: Any) -> SimpleContextManager: - """A context manager tuned to compact aggressively, like the sticky tests.""" - config: dict[str, Any] = { - "max_tokens": 2000, - "compact_threshold": 0.5, - "target_usage": 0.3, - "protected_recent": 0.2, - "protected_tool_results": 1, - "compaction_notice_enabled": False, - } - config.update(overrides) - return SimpleContextManager(**config) - - -def _make_gentle_context(**overrides: Any) -> SimpleContextManager: - """Enough pressure to reach the TRUNCATION rungs (levels 1-2) and stop. - - The aggressive `_make_context` settings escalate all the way to level 8, - where every tool result has been REMOVED rather than truncated -- which - makes any assertion about truncated content vacuously true. Calibrated - against the ladder (measured: 11 truncated results, sticky level 2). - """ - config: dict[str, Any] = { - "max_tokens": 20_000, - "compact_threshold": 0.5, - "target_usage": 0.45, - "protected_recent": 0.2, - "protected_tool_results": 1, - "compaction_notice_enabled": False, - } - config.update(overrides) - return SimpleContextManager(**config) - - -def _tool_msg( - call_id: str, content: str, seq: int | None = 0, **extra: Any -) -> dict[str, Any]: - msg: dict[str, Any] = { - "role": "tool", - "tool_call_id": call_id, - "content": content, - **extra, - } - if seq is not None: - msg["metadata"] = {**(msg.get("metadata") or {}), "_seq": seq} - return msg - - -def _assistant_call(call_id: str, tool_name: str, openai_shape: bool = False) -> dict: - if openai_shape: - return { - "role": "assistant", - "content": "", - "tool_calls": [ - { - "id": call_id, - "type": "function", - "function": {"name": tool_name, "arguments": "{}"}, - } - ], - } - return { - "role": "assistant", - "content": "", - "tool_calls": [{"id": call_id, "tool": tool_name, "arguments": {}}], - } - - -def _legacy_expected(content: str, truncate_chars: int) -> str: - """The exact string this module emitted BEFORE this feature existed. - - Transcribed from the pre-change source, not from the new implementation -- - so this is an independent oracle, not a tautology. - """ - original_tokens = len(content) // 4 - return ( - f"[truncated: ~{original_tokens:,} tokens - call tool again if needed] " - f"{content[:truncate_chars]}..." - ) - - -# -------------------------------------------------------------------------- -# 1. BYTE IDENTITY -- the defaults-are-a-no-op contract -# -------------------------------------------------------------------------- - - -def test_default_config_replacement_text_is_byte_identical(): - """THE byte-identity test. Default config, literal expected bytes.""" - context = SimpleContextManager() - content = "A" * 5000 - out = context._truncate_tool_result(_tool_msg("c1", content)) - - assert out["content"] == _legacy_expected(content, 250) - # And spelled out literally once, so a change to _legacy_expected() alone - # cannot silently re-baseline this test. - assert out["content"].startswith( - "[truncated: ~1,250 tokens - call tool again if needed] AAA" - ) - assert out["content"].endswith("...") - assert len(out["content"]) == len( - "[truncated: ~1,250 tokens - call tool again if needed] " - ) + 250 + 3 - assert out["_truncated"] is True - assert out["_original_tokens"] == 1250 - - -def test_default_config_honors_custom_truncate_chars(): - """A pre-existing `truncate_chars` override must keep working untouched.""" - context = SimpleContextManager(truncate_chars=40) - content = "B" * 900 - out = context._truncate_tool_result(_tool_msg("c1", content)) - assert out["content"] == _legacy_expected(content, 40) - - -def test_default_config_short_result_untouched_and_identical_object(): - context = SimpleContextManager() - msg = _tool_msg("c1", "short") - assert context._truncate_tool_result(msg) is msg - - -def test_default_config_non_string_content_untouched(): - context = SimpleContextManager() - msg = _tool_msg("c1", "") - msg["content"] = [{"type": "text", "text": "x" * 5000}] - assert context._truncate_tool_result(msg) is msg - - -@pytest.mark.asyncio -async def test_default_config_full_compaction_view_is_byte_identical(): - """End-to-end: every truncated result in a real compacted view matches - the pre-change formula exactly.""" - context = _make_gentle_context(truncate_chars=60) - originals: dict[str, str] = {} - for i in range(30): - await context.add_message({"role": "user", "content": f"turn {i} " + "u" * 200}) - await context.add_message(_assistant_call(f"call_{i}", "bash")) - body = f"result {i} " + "z" * 800 - originals[f"call_{i}"] = body - await context.add_message( - {"role": "tool", "tool_call_id": f"call_{i}", "content": body} - ) - - view = await context.get_messages_for_request() - assert context._last_compaction_stats is not None, "test must actually compact" - - truncated_seen = 0 - for msg in view: - if msg.get("role") != "tool": - continue - if not msg.get("_truncated"): - continue - truncated_seen += 1 - original = originals[msg["tool_call_id"]] - assert msg["content"] == _legacy_expected(original, 60) - assert truncated_seen > 0, "test must actually truncate something" - - -@pytest.mark.asyncio -async def test_default_config_writes_no_files_anywhere(tmp_path): - """With no spill dir configured, nothing is ever written to disk.""" - context = _make_gentle_context(truncate_chars=60) - for i in range(30): - await context.add_message({"role": "user", "content": f"t{i} " + "u" * 200}) - await context.add_message(_assistant_call(f"call_{i}", "bash")) - await context.add_message( - {"role": "tool", "tool_call_id": f"call_{i}", "content": f"r{i} " + "z" * 800} - ) - await context.get_messages_for_request() - - assert context._truncated_seqs, "test must actually truncate something" - assert context.tool_result_spill_dir is None - assert context._spilled_paths == set() - assert list(tmp_path.iterdir()) == [] - - -# -------------------------------------------------------------------------- -# 2. TOKEN-DENOMINATED BUDGET -# -------------------------------------------------------------------------- - - -def test_token_budget_keeps_four_chars_per_token(): - context = SimpleContextManager(tool_result_budget_tokens=1000) - content = "C" * 20_000 - out = context._truncate_tool_result(_tool_msg("c1", content)) - body = out["content"].split("] ", 1)[1] - assert body == "C" * 4000 + "..." - - -def test_token_budget_leaves_under_budget_results_alone(): - context = SimpleContextManager(tool_result_budget_tokens=1000) - msg = _tool_msg("c1", "D" * 3999) - assert context._truncate_tool_result(msg) is msg - - -def test_token_budget_header_shape_matches_legacy_head_shape(): - """Switching only the UNIT must not change the FORMAT.""" - context = SimpleContextManager(tool_result_budget_tokens=10) - content = "E" * 900 - out = context._truncate_tool_result(_tool_msg("c1", content)) - assert out["content"] == _legacy_expected(content, 40) - - -# -------------------------------------------------------------------------- -# 3. HEAD + TAIL -- the measured quality gap -# -------------------------------------------------------------------------- - - -def test_head_tail_keeps_the_tail(): - """G-TRB-TAIL, as a unit test: the tail of the ORIGINAL survives.""" - context = SimpleContextManager( - tool_result_budget_tokens=100, tool_result_shape=TOOL_RESULT_SHAPE_HEAD_TAIL - ) - content = "HEAD" + "m" * 10_000 + "FAILED: assertion at line 42" - out = context._truncate_tool_result(_tool_msg("c1", content))["content"] - - assert out.endswith("FAILED: assertion at line 42") - assert "HEAD" in out - assert "...[" in out and "chars omitted]..." in out - - -def test_head_tail_splits_budget_in_half_and_counts_omission_exactly(): - context = SimpleContextManager( - tool_result_budget_tokens=100, tool_result_shape=TOOL_RESULT_SHAPE_HEAD_TAIL - ) - content = "".join(str(i % 10) for i in range(10_000)) - out = context._truncate_tool_result(_tool_msg("c1", content))["content"] - - body = out.split("] ", 1)[1] - head, rest = body.split("\n...[", 1) - omitted_str, tail = rest.split(" chars omitted]...\n", 1) - - assert head == content[:200] - assert tail == content[-200:] - assert omitted_str.replace(",", "") == str(10_000 - 400) - - -def test_head_tail_smallest_budget_does_not_leak_whole_content(): - """content[-0:] is the WHOLE string. Guard against that class of bug.""" - context = SimpleContextManager( - tool_result_budget_tokens=1, tool_result_shape=TOOL_RESULT_SHAPE_HEAD_TAIL - ) - content = "X" * 5000 - out = context._truncate_tool_result(_tool_msg("c1", content))["content"] - assert len(out) < 200, f"replacement should be tiny, got {len(out)} chars" - - -def test_head_tail_under_budget_result_untouched(): - context = SimpleContextManager( - tool_result_budget_tokens=100, tool_result_shape=TOOL_RESULT_SHAPE_HEAD_TAIL - ) - msg = _tool_msg("c1", "Y" * 399) - assert context._truncate_tool_result(msg) is msg - - -# -------------------------------------------------------------------------- -# 4. PER-TOOL BUDGETS AND EXEMPTIONS -# -------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_per_tool_budget_resolved_via_harvested_tool_call_id(): - context = _make_context( - tool_result_budget_tokens=10, - tool_result_budget_by_tool={"grep": 500}, - ) - await context.add_message(_assistant_call("call_g", "grep")) - await context.add_message(_assistant_call("call_b", "bash")) - - grep_result = _tool_msg("call_g", "G" * 9000, seq=None) - bash_result = _tool_msg("call_b", "B" * 9000, seq=None) - - grep_out = context._truncate_tool_result(grep_result)["content"] - bash_out = context._truncate_tool_result(bash_result)["content"] - - assert grep_out.split("] ", 1)[1] == "G" * 2000 + "..." # 500 tokens - assert bash_out.split("] ", 1)[1] == "B" * 40 + "..." # 10 tokens (global) - - -@pytest.mark.asyncio -async def test_per_tool_budget_resolved_via_openai_function_shape(): - context = _make_context(tool_result_budget_by_tool={"read_file": 300}) - await context.add_message(_assistant_call("c1", "read_file", openai_shape=True)) - out = context._truncate_tool_result(_tool_msg("c1", "R" * 9000, seq=None)) - assert out["content"].split("] ", 1)[1] == "R" * 1200 + "..." - - -def test_per_tool_budget_resolved_via_name_field_on_the_result(): - context = SimpleContextManager(tool_result_budget_by_tool={"grep": 5}) - out = context._truncate_tool_result(_tool_msg("c1", "G" * 900, name="grep")) - assert out["content"].split("] ", 1)[1] == "G" * 20 + "..." - - -def test_per_tool_budget_resolved_via_metadata_tool_name(): - context = SimpleContextManager(tool_result_budget_by_tool={"grep": 5}) - msg = _tool_msg("c1", "G" * 900) - msg["metadata"] = {**msg["metadata"], "tool_name": "grep"} - out = context._truncate_tool_result(msg) - assert out["content"].split("] ", 1)[1] == "G" * 20 + "..." - - -def test_unresolvable_tool_name_falls_back_to_the_global_budget(): - context = SimpleContextManager( - tool_result_budget_tokens=10, tool_result_budget_by_tool={"grep": 500} - ) - out = context._truncate_tool_result(_tool_msg("unknown_id", "Z" * 900)) - assert out["content"].split("] ", 1)[1] == "Z" * 40 + "..." - - -def test_per_tool_budget_alone_leaves_other_tools_on_the_legacy_path(): - """Setting ONLY the per-tool map must not change any other tool's bytes.""" - context = SimpleContextManager(tool_result_budget_by_tool={"grep": 500}) - content = "Q" * 900 - out = context._truncate_tool_result(_tool_msg("c1", content, name="bash")) - assert out["content"] == _legacy_expected(content, 250) - - -def test_exempt_tool_is_never_truncated(): - context = SimpleContextManager( - tool_result_budget_tokens=10, tool_result_exempt_tools=["load_skill"] - ) - msg = _tool_msg("c1", "S" * 50_000, name="load_skill") - assert context._truncate_tool_result(msg) is msg - - -@pytest.mark.asyncio -async def test_exempt_tool_survives_full_compaction_pressure(): - """The skill output is still whole after the ladder has run everywhere.""" - skill_body = "SKILL-BODY " + "s" * 800 - # NOTE: protected_tool_results=1, not 0. `tool_result_indices[-0:]` is the - # WHOLE list, so setting 0 protects EVERY tool result from truncation -- - # the exact opposite of what it reads like. Pre-existing behavior, filed - # separately; this test just avoids the trap. - context = _make_gentle_context( - tool_result_budget_tokens=10, - tool_result_exempt_tools=["load_skill"], - protected_tool_results=1, - ) - await context.add_message(_assistant_call("call_skill", "load_skill")) - await context.add_message( - {"role": "tool", "tool_call_id": "call_skill", "content": skill_body} - ) - for i in range(30): - await context.add_message({"role": "user", "content": f"t{i} " + "u" * 200}) - await context.add_message(_assistant_call(f"call_{i}", "bash")) - await context.add_message( - {"role": "tool", "tool_call_id": f"call_{i}", "content": f"r{i} " + "z" * 800} - ) - - view = await context.get_messages_for_request() - assert context._last_compaction_stats is not None, "test must actually compact" - - skill_msgs = [m for m in view if m.get("tool_call_id") == "call_skill"] - if skill_msgs: # may be removed entirely at high pressure -- but never trimmed - assert skill_msgs[0]["content"] == skill_body - assert not skill_msgs[0].get("_truncated") - other_truncated = [ - m - for m in view - if m.get("role") == "tool" and m.get("_truncated") and m["tool_call_id"] != "call_skill" - ] - assert other_truncated, "test must actually truncate the non-exempt results" - - -# -------------------------------------------------------------------------- -# 5. SPILL TO DISK -# -------------------------------------------------------------------------- - - -def test_spill_writes_full_content_and_points_at_it(tmp_path): - context = SimpleContextManager( - tool_result_budget_tokens=50, - tool_result_shape=TOOL_RESULT_SHAPE_HEAD_TAIL, - tool_result_spill_dir=str(tmp_path / "spill"), - ) - content = "HEAD" + "m" * 20_000 + "TAIL" - out = context._truncate_tool_result(_tool_msg("c1", content))["content"] - - written = list((tmp_path / "spill").iterdir()) - assert len(written) == 1 - assert written[0].read_text() == content - assert str(written[0]) in out - assert "read " in out and "for the full result" in out - - -def test_spill_pointer_is_content_addressed_and_idempotent(tmp_path): - context = SimpleContextManager( - tool_result_budget_tokens=50, tool_result_spill_dir=str(tmp_path) - ) - msg = _tool_msg("c1", "N" * 9000) - first = context._truncate_tool_result(msg)["content"] - written = list(tmp_path.iterdir()) - mtime = written[0].stat().st_mtime_ns - - # Re-derive 5 more times, exactly as _apply_sticky_decisions does. - for _ in range(5): - assert context._truncate_tool_result(msg)["content"] == first - assert list(tmp_path.iterdir()) == written - assert written[0].stat().st_mtime_ns == mtime, "spill file was rewritten" - - -def test_spill_paths_differ_for_different_content(tmp_path): - context = SimpleContextManager( - tool_result_budget_tokens=50, tool_result_spill_dir=str(tmp_path) - ) - context._truncate_tool_result(_tool_msg("c1", "P" * 9000, seq=1)) - context._truncate_tool_result(_tool_msg("c2", "Q" * 9000, seq=2)) - assert len(list(tmp_path.iterdir())) == 2 - - -def test_spill_write_failure_still_emits_stable_pointer(tmp_path): - """A failed write must NOT change the emitted bytes. - - Byte-stability outranks pointer validity: a dangling pointer is visible - and recoverable, a silently mutated prefix is a cold cache rebuild. - """ - blocker = tmp_path / "not_a_dir" - blocker.write_text("i am a file") - context = SimpleContextManager( - tool_result_budget_tokens=50, tool_result_spill_dir=str(blocker / "spill") - ) - msg = _tool_msg("c1", "F" * 9000) - - first = context._truncate_tool_result(msg)["content"] - second = context._truncate_tool_result(msg)["content"] - - assert first == second - assert "for the full result" in first - assert not (blocker / "spill").exists() - - -def test_spill_disabled_by_default_writes_nothing(tmp_path): - context = SimpleContextManager(tool_result_budget_tokens=50) - out = context._truncate_tool_result(_tool_msg("c1", "G" * 9000))["content"] - assert "call tool again if needed" in out - assert list(tmp_path.iterdir()) == [] - - -def test_spill_file_content_is_the_untruncated_original(tmp_path): - context = SimpleContextManager( - tool_result_budget_tokens=50, - tool_result_shape=TOOL_RESULT_SHAPE_HEAD_TAIL, - tool_result_spill_dir=str(tmp_path), - ) - content = "\n".join(f"line {i}" for i in range(5000)) - out = context._truncate_tool_result(_tool_msg("c1", content))["content"] - spilled = next(tmp_path.iterdir()) - assert spilled.read_text() == content - # The middle really is missing from the message but present in the file. - assert "line 2500" not in out - assert "line 2500" in spilled.read_text() - - -# -------------------------------------------------------------------------- -# 6. TOOL-PAIR INTEGRITY under the new configuration -# -------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_tool_pairs_stay_atomic_with_every_knob_enabled(tmp_path): - context = _make_gentle_context( - tool_result_budget_tokens=40, - tool_result_shape=TOOL_RESULT_SHAPE_HEAD_TAIL, - tool_result_budget_by_tool={"grep": 10, "bash": 80}, - tool_result_exempt_tools=["load_skill"], - tool_result_spill_dir=str(tmp_path), - ) - for i in range(30): - tool = ["grep", "bash", "load_skill"][i % 3] - await context.add_message({"role": "user", "content": f"t{i} " + "u" * 200}) - await context.add_message(_assistant_call(f"call_{i}", tool)) - await context.add_message( - {"role": "tool", "tool_call_id": f"call_{i}", "content": f"out {i} " + "z" * 800} - ) - - view = await context.get_messages_for_request() - assert context._last_compaction_stats is not None, "test must actually compact" - assert context._truncated_seqs, "test must actually truncate something" - - requested = { - tc["id"] - for m in view - for tc in (m.get("tool_calls") or []) - if isinstance(tc, dict) and tc.get("id") - } - answered = {m["tool_call_id"] for m in view if m.get("role") == "tool"} - assert requested == answered, ( - "tool_use/tool_result atomicity broken: " - f"unanswered={requested - answered} orphaned={answered - requested}" - ) - - -@pytest.mark.asyncio -async def test_exempt_results_are_not_counted_as_truncated(tmp_path): - context = _make_gentle_context( - tool_result_budget_tokens=20, - tool_result_exempt_tools=["load_skill"], - ) - for i in range(30): - await context.add_message({"role": "user", "content": f"t{i} " + "u" * 200}) - await context.add_message(_assistant_call(f"call_{i}", "load_skill")) - await context.add_message( - {"role": "tool", "tool_call_id": f"call_{i}", "content": f"s{i} " + "s" * 800} - ) - - await context.get_messages_for_request() - assert context._truncated_seqs == set(), ( - "an exempt tool result was recorded as truncated -- it would inflate " - "the reported count and pin a sticky decision that does nothing" - ) - - -# -------------------------------------------------------------------------- -# 7. _seq / PREFIX STABILITY -# -------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_prefix_stays_byte_stable_with_head_tail_and_spill(tmp_path): - context = _make_gentle_context( - tool_result_budget_tokens=40, - tool_result_shape=TOOL_RESULT_SHAPE_HEAD_TAIL, - tool_result_spill_dir=str(tmp_path), - ) - for i in range(30): - await context.add_message({"role": "user", "content": f"t{i} " + "u" * 200}) - await context.add_message(_assistant_call(f"call_{i}", "bash")) - await context.add_message( - {"role": "tool", "tool_call_id": f"call_{i}", "content": f"o{i} " + "z" * 800} - ) - - call1 = await context.get_messages_for_request() - assert context._last_compaction_stats is not None, "test must actually compact" - assert context._truncated_seqs, "test must actually truncate something" - assert list(tmp_path.iterdir()), "test must actually spill something" - - await context.add_message({"role": "user", "content": "next turn " + "u" * 200}) - await context.add_message({"role": "assistant", "content": "ack"}) - call2 = await context.get_messages_for_request() - - assert call2[: len(call1)] == call1, ( - "shared prefix changed after appending one turn -- the truncated " - "replacement text (or its spill pointer) is not deterministic" - ) - - -@pytest.mark.asyncio -async def test_truncation_is_idempotent_across_many_requests(tmp_path): - context = _make_gentle_context( - tool_result_budget_tokens=40, - tool_result_shape=TOOL_RESULT_SHAPE_HEAD_TAIL, - tool_result_spill_dir=str(tmp_path), - ) - for i in range(30): - await context.add_message({"role": "user", "content": f"t{i} " + "u" * 200}) - await context.add_message(_assistant_call(f"call_{i}", "bash")) - await context.add_message( - {"role": "tool", "tool_call_id": f"call_{i}", "content": f"o{i} " + "z" * 800} - ) - - baseline = await context.get_messages_for_request() - assert context._truncated_seqs, "test must actually truncate something" - for _ in range(5): - assert await context.get_messages_for_request() == baseline - - # And exactly one spill file per distinct truncated result, no churn. - spilled = sorted(p.name for p in tmp_path.iterdir()) - assert len(spilled) == len(set(spilled)) - assert not any(p.name.endswith(".tmp") for p in tmp_path.iterdir()) - - -@pytest.mark.asyncio -async def test_set_messages_rebuilds_the_per_tool_name_map(): - """A resumed session must resolve per-tool budgets as the original did.""" - context = _make_context(tool_result_budget_by_tool={"grep": 5}) - await context.set_messages( - [ - _assistant_call("call_g", "grep"), - {"role": "tool", "tool_call_id": "call_g", "content": "G" * 900}, - ] - ) - assert context._tool_name_by_call_id == {"call_g": "grep"} - out = context._truncate_tool_result(context.messages[1]) - assert out["content"].split("] ", 1)[1] == "G" * 20 + "..." - - -@pytest.mark.asyncio -async def test_clear_resets_the_per_tool_name_map(): - context = _make_context(tool_result_budget_by_tool={"grep": 5}) - await context.add_message(_assistant_call("call_g", "grep")) - assert context._tool_name_by_call_id - await context.clear() - assert context._tool_name_by_call_id == {} - assert context._spilled_paths == set() - - -@pytest.mark.asyncio -async def test_default_config_does_no_tool_name_harvesting(): - """The default path must not pay for a feature nobody turned on.""" - context = _make_context() - await context.add_message(_assistant_call("call_g", "grep")) - assert context._tool_name_by_call_id == {} - - -# -------------------------------------------------------------------------- -# 8. CONFIG VALIDATION -- warn and fall back, never crash mount() -# -------------------------------------------------------------------------- - - -def test_unknown_shape_falls_back_to_head_with_a_warning(caplog): - with caplog.at_level(logging.WARNING): - context = SimpleContextManager(tool_result_shape="middle_out") - assert context.tool_result_shape == TOOL_RESULT_SHAPE_HEAD - assert "tool_result_shape" in caplog.text - - -@pytest.mark.parametrize("bad", [0, -5, "1000", 3.5, True]) -def test_unusable_budget_tokens_falls_back_to_legacy(bad, caplog): - with caplog.at_level(logging.WARNING): - context = SimpleContextManager(tool_result_budget_tokens=bad) - assert context.tool_result_budget_tokens is None - assert "tool_result_budget_tokens" in caplog.text - content = "A" * 900 - out = context._truncate_tool_result(_tool_msg("c1", content)) - assert out["content"] == _legacy_expected(content, 250) - - -def test_bad_per_tool_entries_are_dropped_individually(caplog): - with caplog.at_level(logging.WARNING): - context = SimpleContextManager( - tool_result_budget_by_tool={"grep": 500, "bash": -1, "ls": "big"} - ) - assert context.tool_result_budget_by_tool == {"grep": 500} - assert "bash" in caplog.text and "ls" in caplog.text - - -def test_non_dict_per_tool_map_is_ignored(caplog): - with caplog.at_level(logging.WARNING): - context = SimpleContextManager(tool_result_budget_by_tool=["grep"]) - assert context.tool_result_budget_by_tool == {} - assert "tool_result_budget_by_tool" in caplog.text - - -def test_exempt_tools_coerced_to_frozenset(): - context = SimpleContextManager(tool_result_exempt_tools=["a", "a", "b"]) - assert context.tool_result_exempt_tools == frozenset({"a", "b"}) - - -def test_defaults_are_all_inert(): - context = SimpleContextManager() - assert context.tool_result_budget_tokens is None - assert context.tool_result_shape == TOOL_RESULT_SHAPE_HEAD - assert context.tool_result_budget_by_tool == {} - assert context.tool_result_exempt_tools == frozenset() - assert context.tool_result_spill_dir is None - - -# -------------------------------------------------------------------------- -# 9. mount() plumbing -# -------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_mount_passes_every_new_flag_through(tmp_path): - from amplifier_module_context_simple import mount - - mounted: dict[str, Any] = {} - - class _Coordinator: - hooks = None - - async def mount(self, slot: str, instance: Any) -> None: - mounted[slot] = instance - - await mount( - _Coordinator(), # type: ignore[arg-type] - { - "tool_result_budget_tokens": 4000, - "tool_result_shape": "head_tail", - "tool_result_budget_by_tool": {"grep": 2000}, - "tool_result_exempt_tools": ["load_skill"], - "tool_result_spill_dir": str(tmp_path), - }, - ) - context = mounted["context"] - assert context.tool_result_budget_tokens == 4000 - assert context.tool_result_shape == TOOL_RESULT_SHAPE_HEAD_TAIL - assert context.tool_result_budget_by_tool == {"grep": 2000} - assert context.tool_result_exempt_tools == frozenset({"load_skill"}) - assert context.tool_result_spill_dir == str(tmp_path) - - -@pytest.mark.asyncio -async def test_mount_defaults_leave_every_flag_inert(): - from amplifier_module_context_simple import mount - - mounted: dict[str, Any] = {} - - class _Coordinator: - hooks = None - - async def mount(self, slot: str, instance: Any) -> None: - mounted[slot] = instance - - await mount(_Coordinator(), {}) # type: ignore[arg-type] - context = mounted["context"] - assert context.tool_result_budget_tokens is None - assert context.tool_result_shape == TOOL_RESULT_SHAPE_HEAD - assert context.tool_result_budget_by_tool == {} - assert context.tool_result_exempt_tools == frozenset() - assert context.tool_result_spill_dir is None - assert context.truncate_chars == 250 - - -def test_spill_dir_is_not_created_until_something_spills(tmp_path): - target = tmp_path / "never" - SimpleContextManager( - tool_result_budget_tokens=50, tool_result_spill_dir=str(target) - ) - assert not target.exists() - - -def test_spill_dir_created_lazily_on_first_spill(tmp_path): - target = tmp_path / "made_on_demand" / "nested" - context = SimpleContextManager( - tool_result_budget_tokens=50, tool_result_spill_dir=str(target) - ) - context._truncate_tool_result(_tool_msg("c1", "H" * 9000)) - assert Path(target).is_dir() - assert len(list(Path(target).iterdir())) == 1 From 2285ad3e5613bd35047ec49079e81e3a482d27d7 Mon Sep 17 00:00:00 2001 From: bkrabach <702425+bkrabach@users.noreply.github.com> Date: Wed, 2 Sep 2026 17:08:25 -0700 Subject: [PATCH 6/7] Revert "feat: token_meter \"hybrid\" -- provider-anchored counts with provenance (+ the never-run estimate-vs-hybrid-vs-actual measurement) (#22)" This reverts commit f47c8948c55f597ed21ff2612dc66aa54ca646ce. Merge policy: main carries wins only. The hybrid token_meter mode's own measurement was never run. Unproven, opt-in feature -- belongs on a branch for evaluation, not on main. --- README.md | 88 +-- amplifier_module_context_simple/__init__.py | 430 +-------------- tests/test_token_meter_hybrid.py | 573 -------------------- 3 files changed, 27 insertions(+), 1064 deletions(-) delete mode 100644 tests/test_token_meter_hybrid.py diff --git a/README.md b/README.md index c2d8055..9a80d85 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ Provides straightforward in-memory conversation context management. This is the - No persistence across sessions - Automatic compaction when approaching token limit (keeps system messages + last 10 messages) - **Preserves tool pairs as atomic units** during compaction (data integrity guarantee) -- **Optional real-usage token meter** (`token_meter: "actual"` or `"hybrid"`, default off) drives the compaction trigger from real provider usage instead of the built-in estimator -- see [Real-usage token meter](#real-usage-token-meter-token_meter) below +- **Optional real-usage token meter** (`token_meter: "actual"`, default off) drives the compaction trigger from real provider usage instead of the built-in estimator -- see [Real-usage token meter](#real-usage-token-meter-token_meter) below ## Configuration @@ -155,92 +155,10 @@ existed. This mirrors context-handoff's own documented limitation that its measurement is retrospective (one-call lag): the meter describes the request that was *just* answered, not the one currently being assembled. -### Hybrid mode (`token_meter: "hybrid"`) - -`"actual"` replaces the estimator wholesale with the provider's last -reported total. That number is **retrospective by one call**: it describes -the request that was just answered, so everything appended since -- the new -user turn, the tool results from this turn's tool loop -- is invisible to it. - -`"hybrid"` is the shape `openai/codex` uses, which has no "estimate mode" at -all: - -``` -total = provider_reported_total_from_the_last_llm_response # kind='usage' - + estimate(items appended since that response) # the un-billed tail only -``` - -The heuristic is still used, but only for the small, recent slice the -provider has not priced yet -- never for the whole window, and never for the -system prompt and tool schemas, which are both the largest block in the -window and the block the provider has *definitely* already billed. - -On top of that shape it carries two things lifted from -`deepseek-ai/deepseek-harness`: - -**1. Provenance on every count (`kind`).** Every number this module produces -reports where it came from -- `'usage'`, `'estimated'`, or `'none'` -- in -`_last_token_meter_stats["kind"]` and on the `context:token_meter` event. -A consumer that acts *irreversibly* on a token count must branch on this -rather than silently accepting an estimate. - -**2. A conservatism guard.** If the provider's total is *below* what the -heuristic priced for the very content it billed, the anchor is not a -trustworthy floor for this window: it is **rejected**, the (larger) full -heuristic is reported instead, and the count is honestly marked -`kind='estimated'`. The comparand is the estimate of the view that was -actually **sent** on that request, not the full uncompacted history -- those -are different numbers whenever compaction is active. - -And a third, from the same source: **refuse to guess.** The optional cache -aggregates in the stats (`cache_aggregates`) are reported **only** when every -usage event observed this session carried the underlying fields. One event -missing them makes the aggregate `None` (undefined) rather than a partial sum -that silently under-reports. - -#### G-METER-PROVENANCE - -In `"hybrid"` mode, **no irreversible action is taken on a count whose kind -is not `'usage'`.** Firing compaction is irreversible in the ways that -matter: it destroys the provider's prompt cache for at least one request, -and it records sticky truncate/remove decisions that persist for the rest of -the session. So when the count is `kind='estimated'` -- before the first -response of the session, or when the conservatism guard rejected the anchor --- the trigger **declines to fire** and waits for provider-anchored data. -Refusals are counted in `_last_token_meter_stats["provenance_refusals"]`. - -There is exactly **one** escape, and it is recorded rather than hidden: if -**no** anchor has ever arrived this session *and* the count has reached 100% -of budget, refusing would guarantee a provider context-overflow failure on -the next request, which is strictly worse than acting on an estimate. That -path fires, logs a warning, and is counted separately in -`provenance_overrides` -- so a gate can report it honestly instead of it -looking like a clean anchored fire. It cannot mask a rejected anchor: it -requires that no measurement exists at all. - -#### All three meters, every request - -`estimated_tokens`, `measured_tokens` and `hybrid_tokens` are computed on -**every** request in **every** mode -- including the default -- and reported -via `_last_token_meter_stats` and the `context:token_meter` event, alongside -`anchor_tokens`, `anchor_rejected`, `tail_estimated_tokens` and -`tail_messages`. Only *which one drives the trigger* changes with the mode. -That is deliberate: it makes the estimate-vs-hybrid-vs-actual divergence -measurable on a real workload without changing behaviour to measure it. - -#### Same sizing limitation as `"actual"` - -Only the **gate** uses the anchored number. `target_tokens` and every -per-level termination check inside `_compact_ephemeral` are still the -estimator, for the same reason as in `"actual"` mode: a provider-billed -count for a hypothetical smaller message set does not exist without another -round trip. - ### Future default flip (pending validation) -`token_meter` defaults to `"estimate"` specifically so it ships -with **zero behavior change**. Flipping the default to `"actual"` or -`"hybrid"` -- and +`token_meter` defaults to `"estimate"` in this PR specifically so it ships +with **zero behavior change**. Flipping the default to `"actual"` -- and potentially raising `compact_threshold` closer to the real ceiling now that it can be measured accurately -- is a follow-up, not part of this change. It should happen only after running the module's own eval harness against diff --git a/amplifier_module_context_simple/__init__.py b/amplifier_module_context_simple/__init__.py index bde6d17..3af9cd3 100644 --- a/amplifier_module_context_simple/__init__.py +++ b/amplifier_module_context_simple/__init__.py @@ -31,27 +31,6 @@ arrived this session (falling back to the estimator before then, or whenever hooks/events are unavailable). Default is `"estimate"`, which keeps behavior byte-identical to before this feature existed. - • Set `token_meter: "hybrid"` to ANCHOR on the provider's own reported - total from the last response and apply the heuristic ONLY to items - appended since that anchor (openai/codex's shape), carrying the - PROVENANCE of the resulting number -- `kind` in - {'usage','estimated','none'} -- on every count - (deepseek-harness's shape). Two guards ride with it: - - CONSERVATISM: if the provider total is below what the heuristic - priced for the same billed content, the anchor is rejected and - the count is honestly marked kind='estimated'. - - REFUSE TO GUESS: optional cache aggregates are reported only when - EVERY usage event this session carried them; otherwise undefined. - G-METER-PROVENANCE: in this mode no irreversible action (the - compaction trigger) is taken on a count that is not kind='usage'. - The single recorded escape -- no anchor has EVER arrived AND the - count has reached 100% of budget, where refusing would guarantee a - provider hard-failure -- fires, logs a warning, and is counted - separately as a provenance OVERRIDE rather than a clean fire. - • ALL THREE meters (estimate / actual / hybrid) are computed on every - request in EVERY mode and reported via `_last_token_meter_stats` and - the `context:token_meter` event, so their divergence is measurable - without changing which one drives the trigger. • Ported from amplifier-module-context-handoff's proven `_on_llm_response` meter. See README "Real-usage token meter" for the full rationale. @@ -118,23 +97,7 @@ # SimpleContextManager._measure_working_tokens. TOKEN_METER_ESTIMATE = "estimate" TOKEN_METER_ACTUAL = "actual" -TOKEN_METER_HYBRID = "hybrid" -_VALID_TOKEN_METERS = ( - TOKEN_METER_ESTIMATE, - TOKEN_METER_ACTUAL, - TOKEN_METER_HYBRID, -) - -# Provenance of a token count (`kind`), reported on EVERY count this module -# produces -- lifted from deepseek-harness's `baseline.kind`. "usage" means -# the number is anchored on the provider's own reported usage; -# "estimated" means it came from the len(str)//4 heuristic (in whole or in -# part, or the anchor was rejected by the conservatism guard); "none" means -# there was nothing to price. G-METER-PROVENANCE: in `hybrid` mode, no -# irreversible action may be taken on a count whose kind is not "usage". -METER_KIND_USAGE = "usage" -METER_KIND_ESTIMATED = "estimated" -METER_KIND_NONE = "none" +_VALID_TOKEN_METERS = (TOKEN_METER_ESTIMATE, TOKEN_METER_ACTUAL) # compaction_strategy config values. "progressive" (default) preserves the # existing truncate/remove ladder exactly (byte-identical -- see module @@ -210,12 +173,8 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = - compaction_notice_verbosity: Notice detail level - "minimal", "normal", "verbose" (default: "normal") - compaction_notice_min_level: Only show notice if compaction level >= this (default: 1) - output_reserve_fraction: Fraction of max_output_tokens to reserve for responses (default: 0.5) - - token_meter: "estimate" (default), "actual" or "hybrid". - "estimate" is byte-identical to pre-existing behavior. - "hybrid" anchors on the provider's own reported total and - estimates only the un-billed tail, carrying provenance - (`kind`) that gates irreversible actions -- see module - docstring. "actual" drives the + - token_meter: "estimate" (default) or "actual". "estimate" is + byte-identical to pre-existing behavior. "actual" drives the compaction trigger from real provider usage (input_tokens + cache_write_tokens, observed via the `llm:response` hook) once at least one response has been observed this session, @@ -395,9 +354,7 @@ def __init__( responses (0.0-1.0, default: 0.5). Lower values give more context budget at the cost of less headroom for long responses. token_meter: "estimate" (default, byte-identical to pre-existing - behavior), "hybrid" (provider-anchored total + heuristic - for the un-billed tail, with provenance gating - irreversible actions), or "actual" (compaction trigger uses real provider + behavior) or "actual" (compaction trigger uses real provider usage from the `llm:response` hook once observed this session -- see module docstring "Real-Usage Token Meter"). An unrecognized value falls back to "estimate" with a @@ -478,31 +435,6 @@ def __init__( # "estimate" mode -- see README "Real-usage token meter". self._last_measured_prompt_tokens: int | None = None self._last_token_meter_stats: dict[str, Any] | None = None - # --- Hybrid meter state (token_meter == "hybrid") --- - # `_anchor_seq` is `_next_seq` frozen at the moment the anchor was - # recorded: every message whose `_seq` is >= it was appended AFTER - # the request the provider billed, so it is exactly the un-billed - # tail the heuristic is still allowed to price (codex's shape). - # `_anchor_estimate` is what the heuristic said about the view that - # was actually SENT on that request -- the comparand for the - # conservatism guard (deepseek's shape). `_last_sent_estimate` is - # the running value that becomes `_anchor_estimate` when the next - # llm:response arrives. - self._anchor_seq: int | None = None - self._anchor_estimate: int | None = None - self._last_sent_estimate: int | None = None - self._last_hybrid_tokens: int | None = None - self._last_hybrid_kind: str = METER_KIND_NONE - # Refuse-to-guess accounting for optional cache aggregates: they are - # reported ONLY when every usage event seen this session carried - # them, else undefined (None). Never partially summed. - self._usage_events: int = 0 - self._usage_events_with_cache: int = 0 - self._usage_cache_read_total: int = 0 - self._usage_cache_write_total: int = 0 - # G-METER-PROVENANCE accounting (observability; see _should_compact). - self._provenance_refusals: int = 0 - self._provenance_overrides: int = 0 self._system_prompt_factory: Callable[[], Awaitable[str]] | None = None # --- Sticky compaction decision state --- @@ -671,52 +603,19 @@ async def get_messages_for_request( # Static mode: use messages as-is (may include stored system messages) working_messages = list(self.messages) - ( - token_count, - meter_source, - estimated_tokens, - meter, - ) = self._measure_working_tokens(working_messages) + token_count, meter_source, estimated_tokens = self._measure_working_tokens( + working_messages + ) self._last_token_meter_stats = { "mode": self.token_meter, "source": meter_source, - # Provenance of `used_tokens` -- present on 100% of counts, in - # every mode (G-METER-PROVENANCE). - "kind": meter["kind"], "used_tokens": token_count, - # All three meters, computed simultaneously on every request, - # regardless of which one is actually driving the trigger. This is - # what makes the estimate-vs-hybrid-vs-actual divergence - # measurable without changing behaviour (POC-05 G-METER-DELTA). "estimated_tokens": estimated_tokens, "measured_tokens": self._last_measured_prompt_tokens, - "hybrid_tokens": meter["hybrid_tokens"], - "hybrid_kind": meter["hybrid_kind"], - "anchor_tokens": meter["anchor_tokens"], - "anchor_estimate": meter["anchor_estimate"], - "anchor_rejected": meter["anchor_rejected"], - "tail_estimated_tokens": meter["tail_estimated_tokens"], - "tail_messages": meter["tail_messages"], - # Undefined (None) unless EVERY usage event this session reported - # cache fields -- refuse to guess, never a partial sum. - "cache_aggregates": self._cache_aggregates(), - "provenance_refusals": self._provenance_refusals, - "provenance_overrides": self._provenance_overrides, "budget": effective_budget, "ratio": (token_count / effective_budget) if effective_budget > 0 else None, } - # Observability emit: lets an eval harness capture all three meters - # per request without patching this module. Never raises, never - # changes the view that is returned. - if self._hooks is not None: - try: - await self._hooks.emit( - "context:token_meter", dict(self._last_token_meter_stats) - ) - except Exception as e: # pragma: no cover - defensive - logger.debug(f"Could not emit context:token_meter event: {e}") - # Summary compaction strategy: trigger an async background # summarization call EARLY (well before compact_threshold, so it has # time to finish -- see module docstring "Summary compaction @@ -726,7 +625,7 @@ async def get_messages_for_request( await self._maybe_trigger_summary_compaction(token_count, effective_budget) # Check if compaction needed (using effective budget with notice reserve deducted) - if self._should_compact(token_count, effective_budget, meter["kind"]): + if self._should_compact(token_count, effective_budget): # Compact EPHEMERALLY - returns new list, working_messages unchanged compacted = await self._compact_ephemeral( effective_budget, working_messages @@ -814,9 +713,9 @@ async def get_messages_for_request( # Strip internal bookkeeping at the module boundary -- everything # above this point (sticky decisions, token accounting) still runs # on messages carrying `_seq`; only what leaves has it removed. - return self._finalize_view(compacted) + return self._strip_internal_metadata(compacted) - return self._finalize_view(working_messages) + return self._strip_internal_metadata(working_messages) # Metadata keys that are internal bookkeeping only and must never cross # the module boundary into a provider-facing view. `_seq` is sticky @@ -890,11 +789,6 @@ async def set_messages(self, messages: list[dict[str, Any]]) -> None: restamped.append({**msg, "metadata": meta}) self.messages = restamped self._next_seq = len(restamped) - # Seqs were just restamped from 0, so any anchor split recorded - # against the OLD numbering is meaningless -- and worse, would - # silently classify restored history as an un-billed tail. Drop it; - # the meter re-anchors on the next llm:response. - self._reset_hybrid_meter_state() self._removed_seqs = set() self._truncated_seqs = set() self._stubbed_seqs = set() @@ -914,7 +808,6 @@ async def clear(self) -> None: self._last_compaction_stats = None self._last_measured_prompt_tokens = None self._last_token_meter_stats = None - self._reset_hybrid_meter_state() self._reset_summary_strategy_state() logger.info("Context cleared") @@ -956,54 +849,10 @@ async def compact(self) -> None: """ pass - def _should_compact( - self, token_count: int, budget: int, kind: str | None = None - ) -> bool: - """Check if context should be compacted. - - `kind` is the PROVENANCE of `token_count` (see METER_KIND_*). It is - only consulted in `token_meter: "hybrid"` mode, where G-METER-PROVENANCE - applies: firing compaction is an irreversible action (it destroys the - provider's prompt cache for at least one request and permanently - records sticky truncate/remove decisions), so it may NOT be taken on a - number the provider never anchored. - - The one deliberate escape, recorded rather than hidden: if NO anchor - has ever arrived this session AND the count has reached 100% of - budget, refusing would guarantee a provider hard-failure on the very - next request, which is strictly worse than acting on an estimate. That - path fires, logs a warning, and is counted separately in - `_provenance_overrides` so a gate reports it honestly instead of it - looking like a clean pass. It cannot mask a guard-rejected anchor: it - requires that no measurement exists at all. - """ + def _should_compact(self, token_count: int, budget: int) -> bool: + """Check if context should be compacted.""" usage = token_count / budget if budget > 0 else 0 should = usage >= self.compact_threshold - if ( - should - and self.token_meter == TOKEN_METER_HYBRID - and kind is not None - and kind != METER_KIND_USAGE - ): - if self._last_measured_prompt_tokens is None and usage >= 1.0: - self._provenance_overrides += 1 - logger.warning( - f"context-simple: token_meter='hybrid' firing compaction on an " - f"UNANCHORED count ({token_count:,} tokens = {usage:.1%} of " - f"budget) because no provider usage has been observed this " - f"session and the count has reached the hard ceiling; " - f"refusing would guarantee a context-overflow failure. " - f"Recorded as a provenance override, not a clean fire." - ) - return True - self._provenance_refusals += 1 - logger.info( - f"context-simple: token_meter='hybrid' REFUSING to fire compaction " - f"on kind={kind!r} ({token_count:,} tokens = {usage:.1%} of budget) " - f"-- G-METER-PROVENANCE: no irreversible action on an un-anchored " - f"number. Waiting for provider-reported usage." - ) - return False if should: logger.info( f"Context at {usage:.1%} capacity ({token_count:,}/{budget:,} tokens), " @@ -1020,19 +869,8 @@ def _exceeds_threshold(self, estimated_tokens: int, budget: int) -> bool: In token_meter="actual" mode with a real measurement available, the REAL measurement decides this, not `estimated_tokens` -- see module docstring "Real-Usage Token Meter" and _measure_working_tokens for - the identical mode/fallback logic. In token_meter="hybrid" mode the - hybrid number decides it, but ONLY when that number is anchored - (kind == 'usage'); an estimated hybrid count falls through to the - estimator branch rather than driving an escalation -- the same - G-METER-PROVENANCE rule _should_compact applies to the outer trigger. - Falls back to `estimated_tokens` in "estimate" mode, or whenever no - real measurement has arrived yet. - - NOTE (unchanged from "actual" mode, and equally true here): only the - GATE uses the anchored number. The SIZING of the reduction -- - target_tokens and every per-level termination check -- is still the - estimator throughout, because a billed token count for a hypothetical - smaller message set does not exist without another round-trip. + the identical mode/fallback logic. Falls back to `estimated_tokens` + in "estimate" mode, or whenever no real measurement has arrived yet. """ if budget <= 0: return False @@ -1041,199 +879,18 @@ def _exceeds_threshold(self, estimated_tokens: int, budget: int) -> bool: and self._last_measured_prompt_tokens is not None ): return (self._last_measured_prompt_tokens / budget) >= self.compact_threshold - if ( - self.token_meter == TOKEN_METER_HYBRID - and self._last_hybrid_tokens is not None - and self._last_hybrid_kind == METER_KIND_USAGE - ): - return (self._last_hybrid_tokens / budget) >= self.compact_threshold return (estimated_tokens / budget) >= self.compact_threshold - def _reset_hybrid_meter_state(self) -> None: - """Drop every hybrid-meter reading. Called from clear() and from - set_messages() (which restamps `_seq` from 0, invalidating any - recorded anchor split). Observability-only in the default - "estimate" mode -- these fields never drive that mode's trigger.""" - self._anchor_seq = None - self._anchor_estimate = None - self._last_sent_estimate = None - self._last_hybrid_tokens = None - self._last_hybrid_kind = METER_KIND_NONE - self._usage_events = 0 - self._usage_events_with_cache = 0 - self._usage_cache_read_total = 0 - self._usage_cache_write_total = 0 - - def _finalize_view(self, messages: list[dict[str, Any]]) -> list[dict[str, Any]]: - """Strip internal metadata and record the heuristic price of the view - that is actually being SENT. - - That recorded number is the conservatism guard's comparand: when the - next `llm:response` arrives, its provider total describes THIS view, - so "is the provider total below what the heuristic would say for the - same content?" is only a meaningful question against the estimate of - the view that was sent -- not against the full, uncompacted history. - """ - view = self._strip_internal_metadata(messages) - self._last_sent_estimate = self._estimate_tokens(view) - return view - - def _cache_aggregates(self) -> dict[str, int] | None: - """Session cache aggregates, or None if UNDEFINED. - - Refuse-to-guess rule, lifted from deepseek-harness's - `deriveTurnTokenUsage`: an optional aggregate is reported only when - EVERY usage event observed this session reported the underlying - fields. One event missing them makes the aggregate undefined -- a - partial sum that silently under-reports is worse than no number. - """ - if self._usage_events == 0: - return None - if self._usage_events_with_cache != self._usage_events: - return None - return { - "events": self._usage_events, - "cache_read_tokens": self._usage_cache_read_total, - "cache_write_tokens": self._usage_cache_write_total, - } - - def _hybrid_split( - self, working_messages: list[dict[str, Any]] - ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: - """Split `working_messages` into (billed prefix, un-billed tail) at - the recorded anchor. - - A message belongs to the tail iff it carries a `_seq` >= `_anchor_seq` - -- i.e. it was appended after the request the provider billed. - Messages with no `_seq` at all (the factory-generated system prompt) - are treated as PREFIX: they were part of the billed request, and - pricing them into the tail on top of an anchor that already contains - them would double-count the single largest block in the window. - """ - if self._anchor_seq is None: - return list(working_messages), [] - prefix: list[dict[str, Any]] = [] - tail: list[dict[str, Any]] = [] - for msg in working_messages: - seq = self._extract_seq(msg) - if seq is not None and seq >= self._anchor_seq: - tail.append(msg) - else: - prefix.append(msg) - return prefix, tail - - def _measure_hybrid( - self, working_messages: list[dict[str, Any]], estimated_tokens: int - ) -> dict[str, Any]: - """Compute the hybrid (provider-anchored + provenance) token count. - - total = provider_reported_total_from_the_last_llm_response - + estimate(items appended since that response) - - This is codex's shape (never price the whole window by heuristic when - the provider has already said what the window cost; heuristic only the - un-billed tail) with deepseek's provenance and conservatism guard - bolted on. - - Returns a dict carrying the number AND its provenance. Computed on - every request in every mode, so the estimate-vs-hybrid-vs-actual - divergence is observable without changing which meter drives the - trigger. - - Guards: - * CONSERVATISM -- if the provider's total is BELOW what the - heuristic priced for the same sent content, the anchor is not - trustworthy as a floor; reject it, report the (larger) full - heuristic, and mark kind='estimated'. Never let a number that may - under-state occupancy authorise an irreversible action. - * REFUSE TO GUESS -- with no anchor at all this is honestly - kind='estimated', not a hybrid number wearing a usage label. - """ - if not working_messages: - return { - "hybrid_tokens": 0, - "hybrid_kind": METER_KIND_NONE, - "anchor_tokens": None, - "anchor_estimate": None, - "anchor_rejected": False, - "tail_estimated_tokens": 0, - "tail_messages": 0, - "reason": "no_messages", - } - - anchor = self._last_measured_prompt_tokens - if anchor is None: - return { - "hybrid_tokens": estimated_tokens, - "hybrid_kind": METER_KIND_ESTIMATED, - "anchor_tokens": None, - "anchor_estimate": None, - "anchor_rejected": False, - "tail_estimated_tokens": None, - "tail_messages": None, - "reason": "no_anchor", - } - - _prefix, tail = self._hybrid_split(working_messages) - tail_estimate = self._estimate_tokens(tail) - - if anchor <= 0: - # A non-positive provider total is not a measurement of anything. - # OBSERVED LIVE, not hypothetical: during this feature's own - # divergence capture a provider returned HTTP 200 with an - # all-zero usage block mid-run (input_tokens=0, output_tokens=0) - # on a ~38k-token request. Trusting that as an anchor would have - # asserted the context was EMPTY. The comparand guard below - # catches it whenever a sent-estimate exists; this catches the - # case where one does not. - return { - "hybrid_tokens": estimated_tokens, - "hybrid_kind": METER_KIND_ESTIMATED, - "anchor_tokens": anchor, - "anchor_estimate": self._anchor_estimate, - "anchor_rejected": True, - "tail_estimated_tokens": tail_estimate, - "tail_messages": len(tail), - "reason": "anchor_non_positive", - } - - if self._anchor_estimate is not None and anchor < self._anchor_estimate: - # Conservatism guard fired: the provider total is smaller than the - # heuristic price of the very content it billed, so it cannot be - # trusted as a floor for this window. - return { - "hybrid_tokens": estimated_tokens, - "hybrid_kind": METER_KIND_ESTIMATED, - "anchor_tokens": anchor, - "anchor_estimate": self._anchor_estimate, - "anchor_rejected": True, - "tail_estimated_tokens": tail_estimate, - "tail_messages": len(tail), - "reason": "anchor_below_heuristic", - } - - return { - "hybrid_tokens": anchor + tail_estimate, - "hybrid_kind": METER_KIND_USAGE, - "anchor_tokens": anchor, - "anchor_estimate": self._anchor_estimate, - "anchor_rejected": False, - "tail_estimated_tokens": tail_estimate, - "tail_messages": len(tail), - "reason": None, - } - def _measure_working_tokens( self, working_messages: list[dict[str, Any]] - ) -> tuple[int, str, int, dict[str, Any]]: - """Return (token_count, source, estimated_tokens, meter) used to - evaluate the compaction trigger this call. + ) -> tuple[int, str, int]: + """Return (token_count, source, estimated_tokens) used to evaluate + the compaction trigger this call. `estimated_tokens` is ALWAYS the len(str)//4 heuristic over - `working_messages` (see _estimate_tokens), and `meter` ALWAYS carries - the hybrid number too -- both computed unconditionally so all three - meters (estimate / actual / hybrid) are observable per request via - `_last_token_meter_stats` regardless of mode. + `working_messages` (see _estimate_tokens) -- computed unconditionally + so the estimator-vs-real-usage drift this meter exists to close is + observable via `_last_token_meter_stats` regardless of mode. `token_count`/`source` are what actually drives `_should_compact`: @@ -1245,41 +902,16 @@ def _measure_working_tokens( `llm:response` (input_tokens + cache_write_tokens -- see `_on_llm_response`), source "measured", if one has arrived this session; otherwise falls back to `estimated_tokens`, source - "estimate". - - token_meter == "hybrid": the provider-anchored total plus a - heuristic price for the un-billed tail, source "hybrid" -- see - `_measure_hybrid`. Its provenance (`kind`) rides along and gates - irreversible actions in `_should_compact`. - - `meter["kind"]` is the provenance of the RETURNED `token_count` (not - of the hybrid number, which is reported separately as `hybrid_kind`) - -- so 100% of counts this module produces carry a provenance, in - every mode. + "estimate" (before the first response of the session, or + whenever hooks/events are unavailable). """ estimated_tokens = self._estimate_tokens(working_messages) - hybrid = self._measure_hybrid(working_messages, estimated_tokens) - self._last_hybrid_tokens = hybrid["hybrid_tokens"] - self._last_hybrid_kind = hybrid["hybrid_kind"] - if ( self.token_meter == TOKEN_METER_ACTUAL and self._last_measured_prompt_tokens is not None ): - meter = {**hybrid, "kind": METER_KIND_USAGE} - return ( - self._last_measured_prompt_tokens, - "measured", - estimated_tokens, - meter, - ) - - if self.token_meter == TOKEN_METER_HYBRID: - meter = {**hybrid, "kind": hybrid["hybrid_kind"]} - return hybrid["hybrid_tokens"], "hybrid", estimated_tokens, meter - - kind = METER_KIND_NONE if not working_messages else METER_KIND_ESTIMATED - meter = {**hybrid, "kind": kind} - return estimated_tokens, "estimate", estimated_tokens, meter + return self._last_measured_prompt_tokens, "measured", estimated_tokens + return estimated_tokens, "estimate", estimated_tokens async def _on_llm_response(self, event: str, data: dict[str, Any]) -> Any: """Hook handler for the canonical `llm:response` event -- records the @@ -1321,20 +953,6 @@ async def _on_llm_response(self, event: str, data: dict[str, Any]) -> Any: if isinstance(input_tokens, int | float): total = int(input_tokens) + int(cache_write_tokens) self._last_measured_prompt_tokens = total - # Hybrid meter bookkeeping (no-op for the trigger unless - # token_meter == "hybrid"; recorded always so the hybrid number - # is observable in every mode -- see _measure_hybrid). - self._anchor_seq = self._next_seq - self._anchor_estimate = self._last_sent_estimate - self._usage_events += 1 - cache_read = usage.get("cache_read_tokens") - cache_write_reported = usage.get("cache_write_tokens") - if isinstance(cache_read, int | float) and isinstance( - cache_write_reported, int | float - ): - self._usage_events_with_cache += 1 - self._usage_cache_read_total += int(cache_read) - self._usage_cache_write_total += int(cache_write_reported) logger.debug( f"context-simple: token_meter recorded real usage from " f"llm:response -- input_tokens={int(input_tokens):,} + " diff --git a/tests/test_token_meter_hybrid.py b/tests/test_token_meter_hybrid.py deleted file mode 100644 index de9c830..0000000 --- a/tests/test_token_meter_hybrid.py +++ /dev/null @@ -1,573 +0,0 @@ -"""Hybrid token meter tests (`token_meter: "hybrid"`). - -The hybrid meter anchors on the provider's OWN reported total from the last -`llm:response` and applies the len(str)//4 heuristic ONLY to items appended -since that anchor (openai/codex's shape), then carries the PROVENANCE of the -resulting number -- `kind` in {'usage','estimated','none'} -- on every count -(deepseek-harness's shape), plus deepseek's conservatism and refuse-to-guess -guards. - -Coverage: - - anchor + un-billed tail is the hybrid number, and the split is by `_seq` - - CONSERVATISM GUARD: an anchor below the heuristic price of the content it - billed is rejected, and the count is honestly marked kind='estimated' - - G-METER-PROVENANCE: an irreversible action (compaction trigger fire) is - REFUSED on kind='estimated', and taken on kind='usage' - - the one recorded escape (no anchor has ever arrived AND the count has hit - the hard ceiling) fires but is counted separately as an override - - REFUSE TO GUESS: cache aggregates are undefined unless EVERY usage event - reported them - - 100% of counts carry a `kind`, in all three modes - - all three meters are computed simultaneously per request in every mode - (the G-METER-DELTA measurement surface) and emitted as - `context:token_meter` - - set_messages() drops a stale anchor split (seqs are restamped from 0) - - "hybrid" is accepted by mount(); unknown values still degrade to - "estimate" -""" - -import pytest -from amplifier_module_context_simple import ( - METER_KIND_ESTIMATED, - METER_KIND_NONE, - METER_KIND_USAGE, - TOKEN_METER_HYBRID, - SimpleContextManager, - mount, -) - -BIG = "lorem ipsum dolor sit amet consectetur " * 20 - - -class _RecordingHooks: - """Minimal hooks stand-in that records register() and emit() calls.""" - - def __init__(self): - self.registered: list[dict] = [] - self.emitted: list[tuple[str, dict]] = [] - - def register(self, event, handler, priority=0, name=None): - entry = {"event": event, "handler": handler, "priority": priority, "name": name} - self.registered.append(entry) - - def unregister(): - self.registered.remove(entry) - - return unregister - - async def emit(self, event, data=None): - self.emitted.append((event, data)) - return None - - -class _FakeCoordinator: - def __init__(self, hooks=None): - self.hooks = hooks - self.mounted: dict[str, object] = {} - - async def mount(self, kind: str, instance: object) -> None: - self.mounted[kind] = instance - - -async def _anchor(context, total: int) -> None: - """Fire a synthetic llm:response carrying `total` as the provider's own - reported prompt usage (input_tokens + cache_write_tokens).""" - await context._on_llm_response( - "llm:response", {"usage": {"input_tokens": total, "cache_write_tokens": 0}} - ) - - -# --------------------------------------------------------------------------- -# The hybrid number: anchor + un-billed tail -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_hybrid_is_provider_anchor_plus_tail_estimate_only(): - """The heuristic prices ONLY what was appended after the anchor -- the - already-billed prefix is the provider's number, not ours.""" - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, token_meter=TOKEN_METER_HYBRID - ) - for i in range(6): - await context.add_message({"role": "user", "content": f"m{i} {BIG}"}) - - # Send once so the meter knows what the heuristic said about the view that - # is about to be billed, then anchor generously above it. - await context.get_messages_for_request() - sent_estimate = context._last_sent_estimate - await _anchor(context, sent_estimate * 4) - - # Two NEW messages arrive after the anchor -- the un-billed tail. - await context.add_message({"role": "user", "content": f"tail-a {BIG}"}) - await context.add_message({"role": "user", "content": f"tail-b {BIG}"}) - tail_estimate = context._estimate_tokens(context.messages[-2:]) - - await context.get_messages_for_request() - stats = context._last_token_meter_stats - - assert stats["hybrid_kind"] == METER_KIND_USAGE - assert stats["anchor_tokens"] == sent_estimate * 4 - assert stats["tail_messages"] == 2 - assert stats["tail_estimated_tokens"] == tail_estimate - assert stats["hybrid_tokens"] == sent_estimate * 4 + tail_estimate - assert stats["used_tokens"] == stats["hybrid_tokens"] - assert stats["source"] == "hybrid" - # The whole point: the hybrid number is NOT the full heuristic. - assert stats["hybrid_tokens"] != stats["estimated_tokens"] - - -@pytest.mark.asyncio -async def test_hybrid_with_no_tail_is_exactly_the_anchor(): - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, token_meter=TOKEN_METER_HYBRID - ) - await context.add_message({"role": "user", "content": BIG}) - await context.get_messages_for_request() - await _anchor(context, context._last_sent_estimate * 3) - - await context.get_messages_for_request() - stats = context._last_token_meter_stats - - assert stats["tail_messages"] == 0 - assert stats["hybrid_tokens"] == stats["anchor_tokens"] - assert stats["kind"] == METER_KIND_USAGE - - -@pytest.mark.asyncio -async def test_hybrid_sums_cache_write_into_the_anchor(): - """Cache-writes are part of context occupancy: a real session reported - input_tokens=2 with cache_write_tokens=161,165 on its first call.""" - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, token_meter=TOKEN_METER_HYBRID - ) - await context.add_message({"role": "user", "content": "hi"}) - await context.get_messages_for_request() - await context._on_llm_response( - "llm:response", {"usage": {"input_tokens": 2, "cache_write_tokens": 161_165}} - ) - - await context.get_messages_for_request() - assert context._last_token_meter_stats["anchor_tokens"] == 161_167 - - -# --------------------------------------------------------------------------- -# Conservatism guard -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_conservatism_guard_rejects_anchor_below_heuristic_price(): - """If the provider total is BELOW what the heuristic priced for the very - content it billed, the anchor is not a trustworthy floor: reject it, - report the larger heuristic, and mark the count kind='estimated'.""" - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, token_meter=TOKEN_METER_HYBRID - ) - for i in range(8): - await context.add_message({"role": "user", "content": f"m{i} {BIG}"}) - await context.get_messages_for_request() - sent_estimate = context._last_sent_estimate - - # Provider says the request cost HALF what the heuristic claimed. - await _anchor(context, sent_estimate // 2) - - await context.get_messages_for_request() - stats = context._last_token_meter_stats - - assert stats["anchor_rejected"] is True - assert stats["hybrid_kind"] == METER_KIND_ESTIMATED - assert stats["kind"] == METER_KIND_ESTIMATED - assert stats["hybrid_tokens"] == stats["estimated_tokens"] - assert stats["anchor_tokens"] == sent_estimate // 2 - assert stats["anchor_estimate"] == sent_estimate - - -@pytest.mark.asyncio -async def test_anchor_at_or_above_heuristic_price_is_accepted(): - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, token_meter=TOKEN_METER_HYBRID - ) - await context.add_message({"role": "user", "content": BIG}) - await context.get_messages_for_request() - await _anchor(context, context._last_sent_estimate) - - await context.get_messages_for_request() - stats = context._last_token_meter_stats - assert stats["anchor_rejected"] is False - assert stats["hybrid_kind"] == METER_KIND_USAGE - - -# --------------------------------------------------------------------------- -# G-METER-PROVENANCE: no irreversible action on kind='estimated' -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_provenance_refuses_compaction_trigger_on_estimated_count(): - """THE gate. In hybrid mode the compaction trigger -- an irreversible - action: it destroys the prompt cache and records sticky truncate/remove - decisions -- must NOT fire on a count the provider never anchored.""" - context = SimpleContextManager( - max_tokens=100_000, - compact_threshold=0.5, - target_usage=0.25, - compaction_notice_enabled=False, - token_meter=TOKEN_METER_HYBRID, - ) - # Grow until the estimator is over threshold but well under the hard - # ceiling, with NO llm:response ever seen -> kind='estimated'. - while context._estimate_tokens(context.messages) < 60_000: - await context.add_message({"role": "user", "content": BIG}) - - await context.get_messages_for_request() - stats = context._last_token_meter_stats - - assert stats["kind"] == METER_KIND_ESTIMATED - assert stats["ratio"] >= 0.5, "gate would be vacuous if not over threshold" - assert stats["ratio"] < 1.0, "must be below the hard-ceiling escape" - assert context._last_compaction_stats is None, ( - "G-METER-PROVENANCE violated: compaction fired on kind='estimated'" - ) - assert context._provenance_refusals == 1 - assert context._provenance_overrides == 0 - - -@pytest.mark.asyncio -async def test_provenance_allows_compaction_trigger_on_anchored_count(): - """The other half of the gate: it must not be vacuous. With the SAME - setup plus a real provider anchor, compaction does fire.""" - context = SimpleContextManager( - max_tokens=100_000, - compact_threshold=0.5, - target_usage=0.25, - compaction_notice_enabled=False, - token_meter=TOKEN_METER_HYBRID, - ) - while context._estimate_tokens(context.messages) < 60_000: - await context.add_message({"role": "user", "content": BIG}) - - await context.get_messages_for_request() # refused (no anchor yet) - assert context._last_compaction_stats is None - await _anchor(context, context._last_sent_estimate + 1) # >= heuristic: accepted - - await context.get_messages_for_request() - stats = context._last_token_meter_stats - - assert stats["kind"] == METER_KIND_USAGE - assert context._last_compaction_stats is not None, ( - "hybrid mode must still compact once the count is provider-anchored" - ) - assert context._provenance_refusals == 1 # the first call only - - -@pytest.mark.asyncio -async def test_provenance_refuses_when_conservatism_guard_rejected_the_anchor(): - """A rejected anchor is kind='estimated' and therefore cannot authorise - the trigger either -- the guard is not a formality.""" - context = SimpleContextManager( - max_tokens=100_000, - compact_threshold=0.5, - target_usage=0.25, - compaction_notice_enabled=False, - token_meter=TOKEN_METER_HYBRID, - ) - while context._estimate_tokens(context.messages) < 60_000: - await context.add_message({"role": "user", "content": BIG}) - - await context.get_messages_for_request() - await _anchor(context, context._last_sent_estimate // 2) # under-states: rejected - - await context.get_messages_for_request() - - assert context._last_token_meter_stats["anchor_rejected"] is True - assert context._last_compaction_stats is None - assert context._provenance_refusals == 2 - assert context._provenance_overrides == 0 - - -@pytest.mark.asyncio -async def test_hard_ceiling_override_fires_but_is_recorded_separately(caplog): - """The one deliberate escape: with NO anchor ever and the count at 100% - of budget, refusing would guarantee a provider hard-failure. It fires -- - and is counted as an override, never as a clean anchored fire.""" - context = SimpleContextManager( - max_tokens=20_000, - compact_threshold=0.5, - target_usage=0.25, - compaction_notice_enabled=False, - token_meter=TOKEN_METER_HYBRID, - ) - while context._estimate_tokens(context.messages) < 21_000: - await context.add_message({"role": "user", "content": BIG}) - - await context.get_messages_for_request() - - assert context._last_token_meter_stats["kind"] == METER_KIND_ESTIMATED - assert context._last_token_meter_stats["ratio"] >= 1.0 - assert context._last_compaction_stats is not None - assert context._provenance_overrides == 1 - assert context._provenance_refusals == 0 - - -@pytest.mark.asyncio -async def test_default_mode_never_refuses_anything(): - """G-METER-PROVENANCE applies to hybrid mode only: the default mode's - trigger is untouched, refusal counters stay at zero.""" - context = SimpleContextManager( - max_tokens=100_000, - compact_threshold=0.5, - target_usage=0.25, - compaction_notice_enabled=False, - ) - while context._estimate_tokens(context.messages) < 60_000: - await context.add_message({"role": "user", "content": BIG}) - - await context.get_messages_for_request() - - assert context._last_compaction_stats is not None - assert context._provenance_refusals == 0 - assert context._provenance_overrides == 0 - - -# --------------------------------------------------------------------------- -# Refuse to guess: optional cache aggregates -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_cache_aggregates_undefined_when_any_event_omitted_them(): - context = SimpleContextManager(token_meter=TOKEN_METER_HYBRID) - await context._on_llm_response( - "llm:response", - {"usage": {"input_tokens": 10, "cache_read_tokens": 5, "cache_write_tokens": 1}}, - ) - assert context._cache_aggregates() is not None - - # One event without the optional fields makes the aggregate undefined -- - # a partial sum that silently under-reports is worse than no number. - await context._on_llm_response("llm:response", {"usage": {"input_tokens": 10}}) - assert context._cache_aggregates() is None - - -@pytest.mark.asyncio -async def test_cache_aggregates_sum_when_every_event_reported_them(): - context = SimpleContextManager(token_meter=TOKEN_METER_HYBRID) - for _ in range(3): - await context._on_llm_response( - "llm:response", - { - "usage": { - "input_tokens": 10, - "cache_read_tokens": 100, - "cache_write_tokens": 7, - } - }, - ) - assert context._cache_aggregates() == { - "events": 3, - "cache_read_tokens": 300, - "cache_write_tokens": 21, - } - - -@pytest.mark.asyncio -async def test_cache_aggregates_undefined_before_any_event(): - assert SimpleContextManager()._cache_aggregates() is None - - -# --------------------------------------------------------------------------- -# Provenance coverage and the three-meter measurement surface -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -@pytest.mark.parametrize("mode", ["estimate", "actual", "hybrid"]) -async def test_every_count_carries_a_kind(mode): - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, token_meter=mode - ) - await context.get_messages_for_request() # empty context - assert context._last_token_meter_stats["kind"] == METER_KIND_NONE - - await context.add_message({"role": "user", "content": BIG}) - await context.get_messages_for_request() - assert context._last_token_meter_stats["kind"] in ( - METER_KIND_USAGE, - METER_KIND_ESTIMATED, - METER_KIND_NONE, - ) - - await _anchor(context, 1_000) - await context.get_messages_for_request() - assert context._last_token_meter_stats["kind"] in ( - METER_KIND_USAGE, - METER_KIND_ESTIMATED, - ) - - -@pytest.mark.asyncio -async def test_all_three_meters_are_computed_in_default_mode(): - """The G-METER-DELTA measurement surface: estimate, actual and hybrid are - all present on every request even in the DEFAULT mode, so the divergence - can be measured without changing which meter drives the trigger.""" - context = SimpleContextManager(max_tokens=1_000_000, compact_threshold=0.99) - await context.add_message({"role": "user", "content": BIG}) - await context.get_messages_for_request() - await _anchor(context, 4_242) - await context.add_message({"role": "user", "content": BIG}) - - await context.get_messages_for_request() - stats = context._last_token_meter_stats - - assert stats["mode"] == "estimate" - assert stats["source"] == "estimate" - assert stats["used_tokens"] == stats["estimated_tokens"] # trigger unchanged - assert stats["measured_tokens"] == 4_242 - assert stats["hybrid_tokens"] == 4_242 + stats["tail_estimated_tokens"] - assert stats["hybrid_kind"] == METER_KIND_USAGE - - -@pytest.mark.asyncio -async def test_token_meter_stats_are_emitted_per_request(): - hooks = _RecordingHooks() - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, hooks=hooks - ) - await context.add_message({"role": "user", "content": BIG}) - await context.get_messages_for_request() - - meter_events = [d for name, d in hooks.emitted if name == "context:token_meter"] - assert len(meter_events) == 1 - for key in ("estimated_tokens", "measured_tokens", "hybrid_tokens", "kind"): - assert key in meter_events[0] - - -@pytest.mark.asyncio -async def test_emit_failure_never_breaks_the_request(): - class _BrokenHooks(_RecordingHooks): - async def emit(self, event, data=None): - raise RuntimeError("hook bus is down") - - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, hooks=_BrokenHooks() - ) - await context.add_message({"role": "user", "content": "hello"}) - view = await context.get_messages_for_request() - assert len(view) == 1 - - -# --------------------------------------------------------------------------- -# Anchor lifecycle -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_set_messages_drops_a_stale_anchor_split(): - """set_messages() restamps every `_seq` from 0, so an anchor split - recorded against the OLD numbering would silently mis-classify restored - history as an un-billed tail. It must be dropped.""" - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, token_meter=TOKEN_METER_HYBRID - ) - for i in range(5): - await context.add_message({"role": "user", "content": f"m{i}"}) - await context.get_messages_for_request() - await _anchor(context, 50_000) - assert context._anchor_seq is not None - - await context.set_messages(await context.get_messages()) - - assert context._anchor_seq is None - assert context._last_measured_prompt_tokens == 50_000 # reading itself survives - await context.get_messages_for_request() - stats = context._last_token_meter_stats - # No anchor split -> everything is prefix, nothing is a tail. - assert stats["tail_messages"] == 0 - - -@pytest.mark.asyncio -async def test_clear_resets_all_hybrid_state(): - context = SimpleContextManager(token_meter=TOKEN_METER_HYBRID) - await context.add_message({"role": "user", "content": BIG}) - await context.get_messages_for_request() - await _anchor(context, 12_345) - - await context.clear() - - assert context._anchor_seq is None - assert context._anchor_estimate is None - assert context._last_sent_estimate is None - assert context._last_measured_prompt_tokens is None - assert context._cache_aggregates() is None - - -# --------------------------------------------------------------------------- -# Config plumbing -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_mount_accepts_hybrid(): - hooks = _RecordingHooks() - coordinator = _FakeCoordinator(hooks) - cleanup = await mount(coordinator, {"token_meter": "hybrid"}) - - context = coordinator.mounted["context"] - assert context.token_meter == TOKEN_METER_HYBRID - assert [e["event"] for e in hooks.registered] == ["llm:response"] - await cleanup() - assert hooks.registered == [] - - -@pytest.mark.asyncio -async def test_unknown_token_meter_still_degrades_to_estimate(caplog): - context = SimpleContextManager(token_meter="hybird") # typo on purpose - assert context.token_meter == "estimate" - - -@pytest.mark.asyncio -async def test_non_positive_anchor_is_never_trusted(): - """A provider total of zero is not a measurement. Observed live during - this feature's own divergence capture: a provider returned HTTP 200 with - an all-zero usage block mid-run on a ~38k-token request. Trusting it - would have asserted the context was empty.""" - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, token_meter=TOKEN_METER_HYBRID - ) - await context.add_message({"role": "user", "content": BIG}) - # No prior send, so there is no comparand for the conservatism guard -- - # this must still be refused on its own merits. - await _anchor(context, 0) - - await context.get_messages_for_request() - stats = context._last_token_meter_stats - - assert stats["anchor_tokens"] == 0 - assert stats["anchor_rejected"] is True - assert stats["hybrid_kind"] == METER_KIND_ESTIMATED - assert stats["hybrid_tokens"] == stats["estimated_tokens"] - - -@pytest.mark.asyncio -async def test_zero_usage_report_after_real_traffic_is_rejected_by_the_guard(): - """The same anomaly, in the shape it was actually observed: several real - responses, then one zero-usage response. The conservatism guard must - reject it rather than let the count collapse to ~zero.""" - context = SimpleContextManager( - max_tokens=1_000_000, compact_threshold=0.99, token_meter=TOKEN_METER_HYBRID - ) - for i in range(4): - await context.add_message({"role": "user", "content": f"m{i} {BIG}"}) - await context.get_messages_for_request() - await _anchor(context, context._last_sent_estimate * 2) - await context.get_messages_for_request() - assert context._last_token_meter_stats["hybrid_kind"] == METER_KIND_USAGE - - await _anchor(context, 0) # provider anomaly - await context.get_messages_for_request() - stats = context._last_token_meter_stats - - assert stats["anchor_rejected"] is True - assert stats["hybrid_kind"] == METER_KIND_ESTIMATED - assert stats["hybrid_tokens"] == stats["estimated_tokens"] From 3b8689d27bf1a9ab59afc822abb9b667ab209d7c Mon Sep 17 00:00:00 2001 From: bkrabach <702425+bkrabach@users.noreply.github.com> Date: Wed, 2 Sep 2026 17:08:38 -0700 Subject: [PATCH 7/7] Revert "feat: compaction_strategy \"summary\" -- LLM rolling-summary compaction (opt-in) (#20)" This reverts commit c6dfbba43a8efb166b07f5255329bb9ed573576a. Merge policy: main carries wins only. The summary compaction strategy's own gate measurement showed a +83% cost regression with summarizer share 26-35%% -- falsified, not a win. Belongs on a branch for evaluation, not on main. --- README.md | 191 ----- amplifier_module_context_simple/__init__.py | 704 +---------------- tests/test_summary_compaction_strategy.py | 791 -------------------- 3 files changed, 1 insertion(+), 1685 deletions(-) delete mode 100644 tests/test_summary_compaction_strategy.py diff --git a/README.md b/README.md index 9a80d85..305c9df 100644 --- a/README.md +++ b/README.md @@ -166,197 +166,6 @@ should happen only after running the module's own eval harness against reduction in compaction cadence (request count / wall time) holds up without a corresponding quality regression. -## Summary compaction strategy (`compaction_strategy`) - -> ### :warning: Opt-in, experimental. Do not enable by default. -> -> Ships behind `compaction_strategy: "progressive"` (default, -> byte-identical to this module's behavior before this feature existed). -> -> A T0/T1 evaluation has now run (n=3 vs. n=5 baselines, S5-CRAC -> scenario). Its two headline results, stated plainly: -> -> - **No retention benefit is demonstrated.** T1 scored 94.0 mean vs. -> T0's 94.4 — statistically and practically indistinguishable, and on a -> metric that is **saturated**: both arms score a perfect 40/40 planted -> constraints and 20/20 post-compaction in *every* run, and have across -> 20+ historical runs. The scenario cannot discriminate retention. This -> is *not* evidence that summaries retain worse — it is the absence of -> evidence either way. A discriminating scenario does not exist yet. -> - **Measured +83% run cost** ($4.73 vs. $2.58) and **+84% compaction -> boundaries** (39.7 vs. 21.6) on a compaction-heavy workload, via a -> boundary-refire loop (see "Known issue" below). This **falsifies** the -> pre-registered prediction that cache economics would land in T0's -> band, in the worse direction. -> -> The mechanism itself is validated and correct (all four pre-registered -> gates pass, including the two the donor design failed catastrophically). -> This flag exists so the strategy can be studied further, not because it -> is known to be better. **Do not enable it by default anywhere.** - -### The idea, and where it comes from - -The progressive ladder above is lossy: once a message is truncated or -removed, that content is gone. `compaction_strategy: "summary"` absorbs the -oldest non-protected span into an LLM-generated rolling summary instead -- -retaining *meaning* at the cost of exact wording, rather than losing the -span outright. - -The IDEAS here -- a structured 5-section summarization prompt, and an -async trigger that fires **early** (well before the hard compaction -threshold, so the LLM call has time to finish off the critical path) -- -are lifted from `amplifier-bundle-context-managed`'s `modules/context-managed/` -rolling summarizer (see that repo's `__init__.py:71-97` for the prompt this -one is adapted from). **All plumbing is rebuilt from scratch** on this -module's own sticky/`_seq` machinery, because a live evaluation of that -donor module *as shipped* found two showstoppers its own 5,890 LOC of tests -never caught: - -1. **It drops a tool call while keeping its result.** Its - `_snap_to_tool_pair_boundary` only checked whether the *immediately - next* message had role `"tool"` -- an adjacency heuristic with no - protected-boundary accounting. In a real multi-turn session this - produced `InvalidRequestError: No tool call found for function call - output` on 29 of 30 turns. -2. **Its summary tiers are `role: "system"`.** The Anthropic provider - hoists every system-role message into the single top-level system - block, so each summary swap rewrote that block and busted the - *system*-prompt cache breakpoint -- not just the conversation-region - one. Measured on a live run: 7 distinct `instructions` hashes across 23 - requests (lengths swinging 44,516 -> 1,113 -> 45,310 tokens), vs. **1** - stable hash for `context-simple`'s own control. - -See `.amplifier/evaluation/treatment-validation/20260901-t4-ctxmanaged/PROBE5-VERDICT.md` -for the full write-up. Neither defect is inherited here: - -- **Tool-pair atomicity**: the absorb boundary is snapped by - `_snap_absorb_boundary`, which reuses the *same* `tool_calls[].id` / - `tool_call_id` identity fields `_check_tool_pair_removable` (above) keys - on -- not an adjacency guess. An assistant `tool_calls` message and every - one of its results are absorbed together, or the whole group is excluded - and left for the next round. Never split. -- **Cache-safe role**: the summary message is `role: "user"`, wrapped in a - `` envelope (so foundation's - `is_real_user_message()` classifies it correctly) -- **never** - `role: "system"`. Unlike the tail compaction notice above, it is **not** - marked `metadata.ephemeral` -- it is meant to persist as stable history. - -### How it's wired into this module's own primitives - -- **Absorption is sticky, not a splice.** Absorbed messages are recorded - via the *existing* `_record_removed()` path -- the exact mechanism - progressive Levels 3/5/7/8 already use -- so `_apply_sticky_decisions()` - replays the absorption byte-identically on every subsequent call. - Candidates are keyed by each message's permanent `_seq`, never by list - index, so there is no "stale boundary" class of bug at all (contrast the - donor's `offset_at_creation` drift-guard, which existed only because its - own design tracked absolute indices in the first place). -- **`self.messages` is still never modified by compaction.** The summary - message is stamped with a fresh `_seq` exactly like `add_message()` - would and *appended* to `self.messages` -- never spliced in at an - earlier position. This keeps this module's core invariant intact, at the - cost of the summary landing wherever `self.messages`' tail happens to be - at swap time (not necessarily immediately after the span it covers) -- - an explicit, disclosed trade-off in exchange for never reordering a - shared, cacheable prefix. -- **Async + fallback.** `summary_trigger` (default `0.60`, an absolute - usage fraction of budget) fires an `asyncio.create_task` off the - critical path, mirroring the donor's early-trigger idea (optionally - driven by the real-usage token meter above when `token_meter: "actual"`). - If the hard compaction threshold is reached and no summary has finished - yet (in flight, failed, timed out, or no provider was ever passed), that - pass falls back to the progressive ladder -- a turn is never blocked and - never fails on a summarizer error. -- **No tier merging in this PR.** Each absorption round produces its own - standalone summary message; a message already carrying - `metadata.type == "context_summary"` is never re-selected for a later - round. `context-managed`'s tier-merging (`_merge_oldest_tiers`) is real - and useful but out of scope here -- see the design mandate for why this - PR keeps scope tight (no custom resume logic, no transcript persistence, - no tool-transcript tool). - -### Configuration - -```toml -[[contexts]] -module = "context-simple" -config = { - compaction_strategy = "summary", # default: "progressive" - summary_trigger = 0.60, # usage fraction that starts the async summarizer - summarization_model = "...", # optional; None uses the provider default - summarization_prompt_path = "...", # optional file override for the 5-section prompt - summarization_timeout_s = 30.0, # provider.complete() timeout before falling back -} -``` - -### What this was built to buy, and what the evaluation actually measured - -**The motivation** was retention: the progressive ladder is lossy, so an -LLM summary that keeps a lossy-but-real account of an absorbed span -*should* retain more than dropping it outright. That was and remains the -reason to build this. It is **not** a cache-cost play — like the -progressive ladder, this strategy still shrinks what the model sees each -turn, which under a grow-only prompt cache is still a cold rebuild of the -shared prefix at the moment of absorption. - -**What the T0/T1 evaluation measured** (n=3 T1 runs vs. n=5 reused T0 -baselines, S5-CRAC scenario, capture root -`.amplifier/evaluation/treatment-validation/20260902-t0t1/`): - -| Gate | Requirement | Result | -|---|---|---| -| **G1 retention** | T1 S5 ≥ T0 band (92–95) | **PASS — but vacuous.** T1 [95, 95, 92] vs. T0 mean 94.4. See note below. | -| **G2 tool-pairs** | zero `InvalidRequestError` | **PASS** — 0 across all runs (the donor design: 29/30 turns failed) | -| **G3 system prompt** | agent `instructions` byte-stable | **PASS** — 1 hash per run, all 3 runs (the `role: "user"` fix holds under a real provider round trip; the donor's agent prompt moved 44,516 → 1,113 → 45,310 within one run) | -| **G4 append-only** | no history re-minting | **PASS** — `ID_ONLY` divergences 0/0/0 | - -> **G1 passed but proves nothing about retention.** `b_constraints` is -> 40/40 and `c_post_compaction` is 20/20 in **every run of both arms**, -> and has been across 20+ historical runs. The only moving part is the -> task score. S5-CRAC is at its ceiling: both strategies already hold -> every planted constraint perfectly, so a "≥ baseline" check against a -> saturated metric is a floor check, not a win. **No retention advantage -> is demonstrated by this evaluation.** Establishing one requires a -> scenario with headroom — one where the progressive baseline measurably -> *loses* constraints. That scenario does not exist yet (TBD). - -Summaries do fire correctly and do carry the material: 63–95 requests per -run carried a summary, all `role: "user"` (zero `role: "system"`), up to -67 messages absorbed in a single round, and the emitted summaries restate -all five planted constraints verbatim. The mechanism works. What is -unproven is that the mechanism *helps*. - -### Known issue: boundary refire roughly doubles compaction cost - -Measured on the same evaluation, and **worse than the pre-registered -prediction** (which expected cache economics in T0's band): - -| Metric | T0 (progressive) | T1 (summary) | Delta | -|---|---:|---:|---:| -| Cache waste | 29.0% | **53.9%** | **+24.9pp** (no overlap between arms) | -| Cache-read share | 0.714 | **0.537** | −0.177 | -| Run cost | $2.58 | **$4.73** | **+83%** | -| Compaction boundaries | 21.6 | **39.7** | **+84%** | - -**Mechanism.** The summarizer itself is only 8–11% of run cost — it is -not the driver. The dominant cost is the near-doubling of compaction -boundaries, and every boundary is a guaranteed cold rebuild against a -grow-only cache. Absorbing a span *shrinks* the request, which pulls -usage back below `summary_trigger` (0.60) sooner, which fires another -absorb/compact cycle sooner: a refire loop. Under an aggressive -compaction config (the evaluation forced `max_tokens: 45000`) the early -trigger — deliberately set well below `compact_threshold` to give the -async call time to finish — drives the extra cycles. - -**Not fixed in this change, tracked honestly.** The obvious levers, none -of which are implemented here: a post-absorb **cooldown** before the -trigger may refire; an **absolute floor** on absorbed-span size so small -absorptions cannot cycle; **hysteresis** on `summary_trigger` (arm at -0.60, disarm only after usage falls below some lower band). Tuning -`summary_trigger` upward is the cheapest first experiment. Any of these -should be validated against the same cost metrics before this flag is -enabled anywhere by default. - ## Dependencies - `amplifier-core>=1.0.0` diff --git a/amplifier_module_context_simple/__init__.py b/amplifier_module_context_simple/__init__.py index 3af9cd3..f5f6396 100644 --- a/amplifier_module_context_simple/__init__.py +++ b/amplifier_module_context_simple/__init__.py @@ -33,57 +33,14 @@ keeps behavior byte-identical to before this feature existed. • Ported from amplifier-module-context-handoff's proven `_on_llm_response` meter. See README "Real-usage token meter" for the full rationale. - -Summary Compaction Strategy (opt-in, default off -- see config -`compaction_strategy`): - • `compaction_strategy: "progressive"` (default) is this module's - existing truncate/remove ladder, completely unchanged -- byte-identical - to before this feature existed. - • `compaction_strategy: "summary"` absorbs the oldest non-protected span - into an LLM-generated rolling summary instead of truncating/removing - it. The IDEAS (structured 5-section prompt, early-async-trigger - design) are lifted from amplifier-bundle-context-managed's rolling - summarizer; ALL plumbing is rebuilt on this module's own sticky/_seq - machinery rather than that donor's index-based splice-and-swap -- see - the "Summary compaction strategy" section in - _select_summary_absorb_seqs/_snap_absorb_boundary/ - _swap_in_pending_summary below for why, and README "Summary - compaction strategy" for the measured donor defects this avoids - (a dropped tool-call/result pair, and a `role: "system"` summary tier - that measurably busted the provider's system-prompt cache breakpoint). - • The summary message is role="user" (never "system"), wrapped in a - `` envelope, and persists as - stable history (not ephemeral, unlike the tail compaction notice). - • MOTIVATED by retention (the progressive ladder is lossy; a summary - keeps a lossy-but-real account of the absorbed span). It is NOT a - cache-cost play: like the progressive ladder, this still shrinks what - the model sees each turn, which under a grow-only cache is still a - cold rebuild at the moment of absorption. - • MEASURED (T0/T1 eval, n=3 vs n=5, S5-CRAC -- see README "Summary - compaction strategy" for the full table): the mechanism is validated - (zero tool-pair errors; agent system prompt byte-stable, 1 hash/run; - append-only) but NO retention benefit is demonstrated -- 94.0 vs 94.4 - on a SATURATED metric (both arms 40/40 constraints, 20/20 - post-compaction, every run). Absence of evidence, not evidence of - parity-by-design; a discriminating scenario does not exist yet. - • KNOWN ISSUE, measured: +83% run cost and +84% compaction boundaries - vs the progressive baseline, via a boundary-refire loop (absorbing a - span shrinks the request below summary_trigger, so it refires - sooner). The summarizer itself is only 8-11% of run cost. Not fixed - here; see README for the candidate levers (cooldown / absolute floor - / trigger hysteresis). OPT-IN, EXPERIMENTAL -- do not enable by - default. """ # Amplifier module metadata __amplifier_module_type__ = "context" -import asyncio -import json import logging from collections.abc import Awaitable, Callable from datetime import UTC, datetime -from pathlib import Path from typing import Any from amplifier_core import ModuleCoordinator @@ -99,60 +56,6 @@ TOKEN_METER_ACTUAL = "actual" _VALID_TOKEN_METERS = (TOKEN_METER_ESTIMATE, TOKEN_METER_ACTUAL) -# compaction_strategy config values. "progressive" (default) preserves the -# existing truncate/remove ladder exactly (byte-identical -- see module -# docstring "Summary compaction strategy"). "summary" opts in to absorbing -# the oldest non-protected span into an LLM-generated rolling summary -# instead of truncating/removing it outright. -COMPACTION_STRATEGY_PROGRESSIVE = "progressive" -COMPACTION_STRATEGY_SUMMARY = "summary" -_VALID_COMPACTION_STRATEGIES = ( - COMPACTION_STRATEGY_PROGRESSIVE, - COMPACTION_STRATEGY_SUMMARY, -) - -# The summary message's envelope source tag and metadata type marker. The -# envelope is what makes foundation's is_real_user_message() classify this -# role="user" message as NOT a real user turn (see module docstring); the -# metadata type marker is how this module recognizes its own past summary -# messages (so they are never re-absorbed into a later summary). -_SUMMARY_ENVELOPE_SOURCE = "context-summary" -_SUMMARY_METADATA_TYPE = "context_summary" - -# Default 5-section summarization prompt, lifted near-verbatim from -# amplifier-bundle-context-managed's modules/context-managed/__init__.py:71-97 -# (the donor's structured summarization prompt -- see README "Summary -# compaction strategy" for full provenance). The donor's two -# `read_transcript` tool references are deliberately dropped: this module -# ships no transcript tool, and pointing an agent at a tool that does not -# exist would be actively misleading. File-overridable via -# `summarization_prompt_path`, mirroring the donor's own -# `summarization_prompt_path` config knob. -DEFAULT_SUMMARIZATION_PROMPT = """\ -Produce a compact summary of the conversation so far. Use the following sections: - -## User Requests & Decisions -List the key requests made by the user and any important decisions reached. - -## Files Examined or Modified -List files that were read, analyzed, or modified during the conversation. - -## Errors Encountered & Resolutions -Describe any errors, failures, or unexpected behavior encountered, and how they were resolved. - -## Current Task State -Describe the current state of work -- what has been completed, what is in progress, and what remains. - -## Key Technical Details -Note any important technical constraints, patterns, configurations, or implementation details -discovered during the conversation. - -## Guidelines -- Be factual and concise. Do not speculate beyond what the conversation contains. -- Preserve numeric values, file paths, error messages, and command outputs exactly. -- Each section may be omitted if there is nothing to report for it. -""" - async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = None): """ @@ -181,29 +84,6 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = falling back to the estimator before then. An unrecognized value falls back to "estimate" with a logged warning rather than crashing mount(). See module docstring. - - compaction_strategy: "progressive" (default) or "summary". See - module docstring "Summary compaction strategy". An - unrecognized value falls back to "progressive" with a logged - warning rather than crashing mount(). - - summary_trigger: Usage fraction (0.0-1.0) at which the summary - strategy starts an async background summarization call, well - ahead of compact_threshold so it has time to finish before - tokens must actually be shed (default: 0.60). Only consulted - when compaction_strategy == "summary". KNOWN ISSUE: because - absorbing a span shrinks the request back below this - fraction, an aggressive (low) trigger refires sooner and - measurably multiplies compaction boundaries -- +84% - boundaries / +83% run cost in the T0/T1 eval. Raising this - is the cheapest lever; see README "Known issue: boundary - refire". - - summarization_model: Model identifier passed to the summarizer's - ChatRequest (default: None, i.e. provider default). - - summarization_prompt_path: Path to a file overriding - DEFAULT_SUMMARIZATION_PROMPT (default: None). - - summarization_timeout_s: Seconds to wait for the summarizer's - provider.complete() call before treating it as a failure and - falling back to progressive compaction for that pass - (default: 30.0). Returns: Cleanup callable that unregisters the token-meter hook (if one was @@ -220,17 +100,6 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = ) token_meter = TOKEN_METER_ESTIMATE - compaction_strategy = config.get( - "compaction_strategy", COMPACTION_STRATEGY_PROGRESSIVE - ) - if compaction_strategy not in _VALID_COMPACTION_STRATEGIES: - logger.warning( - f"context-simple: unknown compaction_strategy {compaction_strategy!r} " - f"(expected one of {_VALID_COMPACTION_STRATEGIES!r}); falling back to " - f"{COMPACTION_STRATEGY_PROGRESSIVE!r}" - ) - compaction_strategy = COMPACTION_STRATEGY_PROGRESSIVE - context = SimpleContextManager( max_tokens=config.get("max_tokens", 200_000), compact_threshold=config.get("compact_threshold", 0.92), @@ -246,11 +115,6 @@ async def mount(coordinator: ModuleCoordinator, config: dict[str, Any] | None = compaction_notice_min_level=config.get("compaction_notice_min_level", 1), output_reserve_fraction=config.get("output_reserve_fraction", 0.5), token_meter=token_meter, - compaction_strategy=compaction_strategy, - summary_trigger=config.get("summary_trigger", 0.60), - summarization_model=config.get("summarization_model"), - summarization_prompt_path=config.get("summarization_prompt_path"), - summarization_timeout_s=config.get("summarization_timeout_s", 30.0), hooks=getattr(coordinator, "hooks", None), ) @@ -327,11 +191,6 @@ def __init__( compaction_notice_min_level: int = 1, output_reserve_fraction: float = 0.5, token_meter: str = TOKEN_METER_ESTIMATE, - compaction_strategy: str = COMPACTION_STRATEGY_PROGRESSIVE, - summary_trigger: float = 0.60, - summarization_model: str | None = None, - summarization_prompt_path: str | None = None, - summarization_timeout_s: float = 30.0, hooks: Any = None, ): """ @@ -359,25 +218,6 @@ def __init__( session -- see module docstring "Real-Usage Token Meter"). An unrecognized value falls back to "estimate" with a logged warning rather than raising. - compaction_strategy: "progressive" (default, byte-identical to - pre-existing behavior) or "summary" -- see module docstring - "Summary compaction strategy". An unrecognized value falls - back to "progressive" with a logged warning rather than - raising. - summary_trigger: Usage fraction (0.0-1.0) at which the summary - strategy kicks off an async background summarization call. - Only consulted when compaction_strategy == "summary". - KNOWN ISSUE (measured): a low trigger refires soon after - each absorption shrinks the request, multiplying - compaction boundaries (+84%) and run cost (+83%) -- see - module docstring and README "Known issue: boundary - refire". - summarization_model: Model identifier for the summarizer's own - ChatRequest. None uses the provider's default model. - summarization_prompt_path: Path to a file overriding - DEFAULT_SUMMARIZATION_PROMPT. None uses the built-in prompt. - summarization_timeout_s: Seconds to wait for the summarizer's - provider.complete() call before treating it as a failure. hooks: Optional hooks instance for emitting observability events and (always, when present) recording real usage for the token meter via `llm:response` -- see `_on_llm_response`. @@ -402,28 +242,8 @@ def __init__( ) token_meter = TOKEN_METER_ESTIMATE self.token_meter = token_meter - if compaction_strategy not in _VALID_COMPACTION_STRATEGIES: - logger.warning( - f"context-simple: unknown compaction_strategy {compaction_strategy!r} " - f"(expected one of {_VALID_COMPACTION_STRATEGIES!r}); falling back to " - f"{COMPACTION_STRATEGY_PROGRESSIVE!r}" - ) - compaction_strategy = COMPACTION_STRATEGY_PROGRESSIVE - self.compaction_strategy = compaction_strategy - self.summary_trigger = summary_trigger - self.summarization_model = summarization_model - self.summarization_prompt_path = summarization_prompt_path - self.summarization_timeout_s = summarization_timeout_s self._hooks = hooks self._last_compaction_stats: dict[str, Any] | None = None - # --- Summary compaction strategy state (compaction_strategy == "summary") --- - # Unused, and never touched, in the default "progressive" mode. - self._cached_provider: Any = None - self._is_summarizing: bool = False - self._pending_summary: dict[str, Any] | None = None - self._summarization_failures: int = 0 - self._summarization_task: "asyncio.Task[None] | None" = None - self._summary_absorbed_count: int = 0 # Real-usage token meter state (see _on_llm_response / # _measure_working_tokens). `_last_measured_prompt_tokens` holds the # most recent real usage observed via `llm:response` @@ -549,13 +369,6 @@ async def get_messages_for_request( """ budget = self._calculate_budget(token_budget, provider) - # Summary compaction strategy needs a provider handle to call the - # summarizer -- cache the latest one seen (mirrors how the donor - # module caches it, context-managed:365-367). No-op in the default - # "progressive" mode. - if self.compaction_strategy == COMPACTION_STRATEGY_SUMMARY and provider is not None: - self._cached_provider = provider - # Reserve token budget for potential compaction notice (if enabled) effective_budget = budget if self.compaction_notice_enabled: @@ -616,14 +429,6 @@ async def get_messages_for_request( "ratio": (token_count / effective_budget) if effective_budget > 0 else None, } - # Summary compaction strategy: trigger an async background - # summarization call EARLY (well before compact_threshold, so it has - # time to finish -- see module docstring "Summary compaction - # strategy" and _maybe_trigger_summary_compaction). No-op in the - # default "progressive" mode. Never raises, never blocks this turn. - if self.compaction_strategy == COMPACTION_STRATEGY_SUMMARY: - await self._maybe_trigger_summary_compaction(token_count, effective_budget) - # Check if compaction needed (using effective budget with notice reserve deducted) if self._should_compact(token_count, effective_budget): # Compact EPHEMERALLY - returns new list, working_messages unchanged @@ -794,7 +599,6 @@ async def set_messages(self, messages: list[dict[str, Any]]) -> None: self._stubbed_seqs = set() self._sticky_level = 0 self._last_compaction_stats = None - self._reset_summary_strategy_state() logger.info(f"Restored {len(messages)} messages to context") async def clear(self) -> None: @@ -808,29 +612,8 @@ async def clear(self) -> None: self._last_compaction_stats = None self._last_measured_prompt_tokens = None self._last_token_meter_stats = None - self._reset_summary_strategy_state() logger.info("Context cleared") - def _reset_summary_strategy_state(self) -> None: - """Reset all `compaction_strategy == "summary"` state -- called from - both set_messages() and clear() so a resumed/cleared session never - carries stale in-flight summarization state across the reset. A - no-op in the default "progressive" mode (the fields are simply - never populated in the first place). - - Cancels any in-flight background summarization task rather than - leaving it to run against a context that has just been reset out - from under it. - """ - if self._summarization_task is not None and not self._summarization_task.done(): - self._summarization_task.cancel() - self._cached_provider = None - self._is_summarizing = False - self._pending_summary = None - self._summarization_failures = 0 - self._summarization_task = None - self._summary_absorbed_count = 0 - async def should_compact(self) -> bool: """Check if context should be compacted. @@ -1091,22 +874,6 @@ async def _compact_ephemeral( msg for msg in messages_to_compact if msg.get("role") != "system" ] - # Summary compaction strategy: if a background summarization call - # has completed since the last escalation, absorb its span NOW -- - # before sticky decisions are (re-)applied, so the new removals and - # the new summary message are both visible to this call's - # _apply_sticky_decisions() below. No-op (did_summary_swap stays - # False) in the default "progressive" mode, and a no-op whenever - # compaction_strategy == "summary" but nothing is pending yet. See - # module docstring "Summary compaction strategy". - did_summary_swap = False - if self.compaction_strategy == COMPACTION_STRATEGY_SUMMARY and ( - self._pending_summary is not None - ): - non_system_messages, did_summary_swap = await self._swap_in_pending_summary( - non_system_messages - ) - # UNITS CONVENTION (see the block comment below): every "are we under # target yet?" comparison in this method and its helpers is TOTAL vs # TOTAL. System messages are extracted from `working_messages` but are @@ -1191,7 +958,7 @@ async def _compact_ephemeral( # sizing of that escalation is only as good as the estimator was # before this meter existed. needs_escalation = self._exceeds_threshold(current_tokens, budget) - if not needs_escalation and not did_summary_swap: + if not needs_escalation: # Sticky state alone already keeps us under the threshold that # triggered compaction in the first place -- nothing NEW needs # deciding this call. Return the already-decided view unchanged; @@ -1205,31 +972,6 @@ async def _compact_ephemeral( ) return final_messages - if not needs_escalation: - # Summary swap alone (see above) already brought us back under - # threshold this call -- no progressive level is needed. Still - # route through _finalize_compaction_with_stats (rather than the - # cheap early-return above) so _last_compaction_stats/hooks - # observe that something DID change this call. - # max_level_reached=0 records "no progressive level was needed". - logger.info( - f"Summary compaction alone reached target: {old_count} raw messages, " - f"{old_tokens:,} raw tokens -> {len(working_messages)} messages, " - f"{current_tokens:,} tokens" - ) - return await self._finalize_compaction_with_stats( - working_messages, - system_messages, - old_count, - old_tokens, - 0, - 0, - 0, - 0, - budget, - target_tokens, - ) - logger.info( f"Compacting context (new escalation): {old_count} raw messages, {old_tokens:,} raw tokens " f"-> {len(working_messages)} messages, {current_tokens:,} tokens after sticky state " @@ -2005,11 +1747,6 @@ async def _finalize_compaction_with_stats( "protected_recent": self.protected_recent, "protected_tool_results": self.protected_tool_results, } - if self.compaction_strategy == COMPACTION_STRATEGY_SUMMARY: - # Only surfaced in "summary" mode -- keeps the default - # "progressive" mode's stats dict shape byte-identical to - # before this feature existed. - stats["messages_absorbed_by_summary"] = self._summary_absorbed_count self._last_compaction_stats = stats # Emit event if hooks available @@ -2155,445 +1892,6 @@ def _format_affected_items(self, level: int, stats: dict[str, Any]) -> str: "- If context is critical, consider asking user to clarify their current goal" ) - # ------------------------------------------------------------------ - # Summary compaction strategy (`compaction_strategy: "summary"`) - # ------------------------------------------------------------------ - # - # Opt-in alternative to the progressive truncate/remove ladder above. - # Lifts the IDEAS from amplifier-bundle-context-managed's rolling - # summarizer -- the structured 5-section prompt, the early-async-trigger - # design -- but rebuilds ALL plumbing on this module's own sticky/_seq - # primitives instead of that donor module's index-based splice-and-swap: - # - # - absorbed messages are recorded via _record_removed(), the SAME - # mechanism progressive Levels 3/5/7/8 already use, so - # _apply_sticky_decisions() replays the absorption byte-identically - # on every subsequent call. Candidates are keyed by each message's - # permanent `_seq`, never by list index/offset, so there is no - # "stale boundary" class of bug here at all -- contrast the donor's - # `offset_at_creation` drift-guard, which existed only because ITS - # design tracked absolute indices in the first place. - # - the summary itself is stamped with a fresh `_seq` exactly like - # add_message() would (see _make_summary_message), and is APPENDED - # to self.messages -- never spliced in -- so self.messages remains - # a strict, append-only log and this module's "compaction never - # modifies self.messages" invariant is never violated. - # - the summary is role="user" (never "system"), wrapped in a - # envelope, and is - # NOT marked ephemeral -- it is meant to persist as stable history, - # unlike the (ephemeral, tail-only) compaction notice above. A - # role="system" summary tier is what measurably busted the donor's - # own provider-level system-prompt cache breakpoint (see README - # "Summary compaction strategy" for the measured numbers); this fix - # is non-negotiable, not cosmetic. - # - the absorb boundary is snapped (_snap_absorb_boundary) so an - # assistant tool_calls message and every one of its tool results - # are absorbed together or not at all -- reusing the same - # tool_call_id identity fields _check_tool_pair_removable already - # keys on, not the donor's adjacent-index-only heuristic (the exact - # gap that let the donor split a live call/result pair in practice). - - def _get_summarization_prompt(self) -> str: - """Return the summarization prompt. - - Reads from `summarization_prompt_path` if configured and the file - exists, mirroring the donor's own file-override knob. Falls back to - DEFAULT_SUMMARIZATION_PROMPT on any OSError, logging a warning -- - never raises. - """ - if self.summarization_prompt_path: - try: - return Path(self.summarization_prompt_path).read_text() - except OSError as e: - logger.warning( - f"context-simple: could not read summarization_prompt_path " - f"{self.summarization_prompt_path!r}: {e}; falling back to " - "the built-in DEFAULT_SUMMARIZATION_PROMPT" - ) - return DEFAULT_SUMMARIZATION_PROMPT - - def _format_messages_for_summarization( - self, messages: list[dict[str, Any]] - ) -> str: - """Format messages into the plain-text transcript the summarizer - reads. Each message becomes '[role]: content'; tool results are - linked back to their call via '[tool_result for {tool_call_id}]: - ...'; tool_calls are rendered as '[tool_call: name(args)]' with - arguments truncated to 500 chars. Adapted from the donor's - `_format_messages_for_summarization`. - """ - lines: list[str] = [] - for msg in messages: - role = msg.get("role", "unknown") - content = msg.get("content", "") - if isinstance(content, list): - parts = [] - for block in content: - if isinstance(block, dict): - text = block.get("text", "") - if text: - parts.append(text) - elif hasattr(block, "text"): - parts.append(block.text) - content = "\n".join(parts) - - if role == "tool": - tc_id = msg.get("tool_call_id", "") - line = f"[tool_result for {tc_id}]: {content}" - else: - line = f"[{role}]: {content}" - - extra_lines: list[str] = [] - for tc in msg.get("tool_calls") or []: - if not isinstance(tc, dict): - continue - if "function" in tc: - name = tc["function"].get("name", "unknown_tool") - raw_args = tc["function"].get("arguments", "{}") - else: - name = tc.get("name") or tc.get("tool", "unknown_tool") - raw_args = tc.get("input") or tc.get("arguments") or {} - if isinstance(raw_args, str): - arg_str = raw_args - else: - arg_str = json.dumps(raw_args, separators=(",", ":")) - if len(arg_str) > 500: - arg_str = arg_str[:500] + "..." - extra_lines.append(f" [tool_call: {name}({arg_str})]") - - lines.append("\n".join([line, *extra_lines]) if extra_lines else line) - - return "\n\n".join(lines) - - def _extract_text_from_response(self, response: Any) -> str: - """Join every text-bearing content block in a ChatResponse. Blocks - without a `.text` attribute (tool calls, thinking, etc.) are - silently skipped.""" - parts = [] - for block in getattr(response, "content", None) or []: - text = getattr(block, "text", None) - if text: - parts.append(text) - return "".join(parts) - - async def _maybe_trigger_summary_compaction( - self, token_count: int, effective_budget: int - ) -> None: - """Start an async background summarization call if usage has - crossed `summary_trigger` and nothing is already in flight/pending. - - Mirrors the donor's early-trigger design (default 0.60, well ahead - of compact_threshold) so the LLM call has time to finish before - tokens must actually be shed -- see _swap_in_pending_summary, which - performs the actual absorption once this completes. Never raises, - never blocks this turn: the provider call itself happens inside an - asyncio.create_task, off the critical path. - """ - if self._is_summarizing or self._pending_summary is not None: - return - if self._cached_provider is None: - logger.debug( - "context-simple: skipping summary trigger -- no cached " - "provider yet (first get_messages_for_request() of the " - "session hasn't run, or the caller never passes one)" - ) - return - if effective_budget <= 0: - return - - usage_fraction = token_count / effective_budget - if usage_fraction < self.summary_trigger: - return - - target_tokens = int(effective_budget * self.target_usage) - if token_count <= target_tokens: - return - excess_tokens = token_count - target_tokens - - seqs = self._select_summary_absorb_seqs(excess_tokens) - if not seqs: - logger.debug( - "context-simple: summary trigger fired but nothing eligible " - "to absorb yet (too little non-protected history)" - ) - return - - self._is_summarizing = True - self._summarization_task = asyncio.create_task( - self._run_summary_compaction_task(seqs) - ) - - def _select_summary_absorb_seqs(self, excess_tokens: int) -> list[int] | None: - """Select a prefix of the oldest, still-live, non-protected, - non-system messages to summarize, sized to shed roughly - `excess_tokens`, snapped so a tool_calls/tool-result pair is never - split (_snap_absorb_boundary). Returns the ordered list of `_seq` - ids to absorb, or None if nothing qualifies. - - Excludes: system messages (never compacted, handled separately), - messages already absorbed/removed by a prior escalation, and this - module's own past summary messages (never re-summarized -- each - escalation produces its own standalone summary; see module - docstring for why this PR does not implement tier merging). - """ - live = [ - m - for m in self.messages - if m.get("role") != "system" - and self._extract_seq(m) not in self._removed_seqs - and (m.get("metadata") or {}).get("type") != _SUMMARY_METADATA_TYPE - ] - if not live: - return None - - last_user_idx = None - for i, m in enumerate(live): - if m.get("role") == "user": - last_user_idx = i - - protected_boundary = int(len(live) * (1 - self.protected_recent)) - if last_user_idx is not None: - protected_boundary = min(protected_boundary, last_user_idx) - if protected_boundary <= 0: - return None - - accumulated = 0 - end_idx = 0 - for i in range(protected_boundary): - accumulated += len(str(live[i])) // 4 - end_idx = i + 1 - if accumulated >= excess_tokens: - break - - end_idx = self._snap_absorb_boundary(live, end_idx, protected_boundary) - if end_idx <= 0: - return None - - seqs = [self._extract_seq(m) for m in live[:end_idx]] - return [s for s in seqs if s is not None] or None - - def _snap_absorb_boundary( - self, - live: list[dict[str, Any]], - end_idx: int, - protected_boundary: int, - ) -> int: - """Adjust `end_idx` (an exclusive boundary into `live`) so an - assistant tool_calls message and every one of its tool results are - absorbed together, or not absorbed at all -- and so the boundary - never crosses into the protected tail (index >= protected_boundary). - - Reuses the same identity fields _check_tool_pair_removable keys on - (`tool_calls[].id` / `tool_call_id`), applied to a single - contiguous prefix boundary instead of scattered removal candidates. - This is the fix for the donor's exact production failure: its - `_snap_to_tool_pair_boundary` only checked whether the immediately - NEXT message had role "tool" -- an adjacency heuristic that misses - non-adjacent results and does no protected-boundary accounting at - all, which is how it shipped dropping a `function_call` while - keeping its `function_call_output` (InvalidRequestError, see - README "Summary compaction strategy"). - """ - if end_idx <= 0: - return 0 - - id_map: dict[str, list[int]] = {} - for idx, msg in enumerate(live): - tcid = msg.get("tool_call_id") - if tcid: - id_map.setdefault(tcid, []).append(idx) - - def result_indices(assistant_msg: dict[str, Any]) -> list[int]: - idxs: list[int] = [] - for tc in assistant_msg.get("tool_calls") or []: - tc_id = tc.get("id") or tc.get("tool_call_id") - if tc_id: - idxs.extend(id_map.get(tc_id, [])) - return idxs - - # Bounded fixed-point: each iteration either grows end_idx (capped - # at protected_boundary) or shrinks it to exclude exactly one - # unabsorbable call -- never both for the same call twice -- so - # this always terminates within len(live) iterations. - for _ in range(len(live) + 1): - max_needed = end_idx - overflow_call_idx: int | None = None - for i in range(end_idx): - msg = live[i] - if msg.get("role") == "assistant" and msg.get("tool_calls"): - for k in result_indices(msg): - if k >= max_needed: - max_needed = k + 1 - if k >= protected_boundary and overflow_call_idx is None: - overflow_call_idx = i - if max_needed <= protected_boundary: - return max_needed - # Can't extend past the protected tail without splitting a - # pair -- drop the first offending call (and, transitively, - # everything after it in this round) rather than ever crossing - # the boundary or absorbing a call without its result. - end_idx = overflow_call_idx if overflow_call_idx is not None else 0 - if end_idx <= 0: - return 0 - return 0 # defensive; unreachable given the termination argument above - - async def _run_summary_compaction_task(self, seqs: list[int]) -> None: - """Background task: call the summarizer over the message span - identified by `seqs` and stash the result in `_pending_summary` for - the next get_messages_for_request()/_compact_ephemeral() call to - swap in (_swap_in_pending_summary). - - Never raises: any failure (bad/absent provider, timeout, malformed - response, empty summary) increments `_summarization_failures`, - logs a warning, and leaves `_pending_summary` unset -- the next - compaction pass that needs to shed tokens falls back to the - progressive ladder for that pass, exactly as if - compaction_strategy were "progressive". Always clears - `_is_summarizing`/`_summarization_task` in a finally block so a - failed round never permanently wedges future triggers. - """ - try: - seq_set = set(seqs) - messages_to_summarize = [ - m for m in self.messages if self._extract_seq(m) in seq_set - ] - if not messages_to_summarize: - return - - provider = self._cached_provider - if provider is None: - raise RuntimeError("no cached provider available for summary compaction") - - prompt = self._get_summarization_prompt() - formatted = self._format_messages_for_summarization(messages_to_summarize) - - from amplifier_core import ChatRequest, Message - - request = ChatRequest( - messages=[ - Message(role="system", content=prompt), - Message(role="user", content=formatted), - ], - model=self.summarization_model, - ) - - if self._hooks is not None: - try: - await self._hooks.emit( - "context:pre_summarize", - {"message_count": len(messages_to_summarize)}, - ) - except Exception as e: - logger.warning(f"Could not emit context:pre_summarize: {e}") - - response = await asyncio.wait_for( - provider.complete(request), timeout=self.summarization_timeout_s - ) - summary_text = self._extract_text_from_response(response) - if not summary_text.strip(): - raise ValueError("summarizer returned empty text") - - self._pending_summary = {"seqs": frozenset(seqs), "text": summary_text} - self._summarization_failures = 0 - - if self._hooks is not None: - try: - await self._hooks.emit( - "context:post_summarize", - {"summary_length": len(summary_text)}, - ) - except Exception as e: - logger.warning(f"Could not emit context:post_summarize: {e}") - except Exception as e: - self._summarization_failures += 1 - logger.warning( - f"context-simple: summary compaction failed ({e!r}); falling " - "back to progressive compaction for the next pass that needs " - "to shed tokens" - ) - finally: - self._is_summarizing = False - self._summarization_task = None - - def _make_summary_message(self, summary_text: str) -> dict[str, Any]: - """Build the persisted summary message: role="user" (never - "system" -- see module docstring), wrapped in a - envelope so - foundation's is_real_user_message() classifies it correctly, and - stamped with a fresh `_seq` exactly like add_message() would. - - NOT marked metadata.ephemeral=True: unlike the tail compaction - notice, this message is meant to persist as stable history. - """ - content = ( - f'\n' - f"{summary_text}\n" - "" - ) - message: dict[str, Any] = { - "role": "user", - "content": content, - "metadata": { - "timestamp": datetime.now(UTC).isoformat(timespec="milliseconds"), - "type": _SUMMARY_METADATA_TYPE, - "_seq": self._next_seq, - }, - } - self._next_seq += 1 - return message - - async def _swap_in_pending_summary( - self, non_system_messages: list[dict[str, Any]] - ) -> tuple[list[dict[str, Any]], bool]: - """Absorb a completed pending summary, if it is still valid. - - Validity is checked purely by `_seq` membership -- every captured - seq must still be present in `non_system_messages` and not already - removed by an intervening escalation. There is no index/offset - arithmetic here, so there is no "stale boundary" bug to guard - against beyond this membership check; if the whole span was - already resolved (e.g. an emergency progressive escalation ran in - between), the summary is discarded gracefully -- NOT counted as a - failure -- and the caller falls back to progressive compaction for - this pass, same as if none had been pending. - - Never hand-splices self.messages: absorbed messages are recorded - via _record_removed() (the existing sticky-decision path), and the - summary message is APPENDED to self.messages, never inserted at an - arbitrary position. - - Returns (possibly-updated non_system_messages, did_swap). - """ - pending = self._pending_summary - self._pending_summary = None - - present_seqs = {self._extract_seq(m) for m in non_system_messages} - absorb_seqs = { - s - for s in pending["seqs"] - if s in present_seqs and s not in self._removed_seqs - } - if not absorb_seqs: - logger.info( - "context-simple: discarding stale pending summary -- its " - "absorbed span was already resolved by an earlier escalation" - ) - return non_system_messages, False - - for msg in non_system_messages: - if self._extract_seq(msg) in absorb_seqs: - self._record_removed(msg) - - summary_message = self._make_summary_message(pending["text"]) - self.messages.append(summary_message) - self._summary_absorbed_count += len(absorb_seqs) - - logger.info( - f"context-simple: summary compaction absorbed {len(absorb_seqs)} " - "messages into 1 stable summary message" - ) - return non_system_messages + [summary_message], True - def _calculate_budget(self, token_budget: int | None, provider: Any | None) -> int: """Calculate effective token budget from provider or fallback to config. diff --git a/tests/test_summary_compaction_strategy.py b/tests/test_summary_compaction_strategy.py deleted file mode 100644 index c7a9397..0000000 --- a/tests/test_summary_compaction_strategy.py +++ /dev/null @@ -1,791 +0,0 @@ -"""Adversarial tests for `compaction_strategy: "summary"`. - -This strategy lifts IDEAS from amplifier-bundle-context-managed's rolling -summarizer (the structured 5-section prompt, the early-async-trigger design) -but rebuilds ALL plumbing on this module's own sticky/_seq machinery. The -donor module's own 5,890 LOC of tests missed two showstoppers found by -actually running it (see .amplifier/evaluation/treatment-validation/ -20260901-t4-ctxmanaged/PROBE5-VERDICT.md): - - 1. It drops a `function_call` while keeping its `function_call_output` - (tool-pair atomicity violation) -- `_snap_to_tool_pair_boundary` only - checked adjacency, not actual id-based pairing, and did no - protected-boundary accounting at all. - 2. Its summary tiers are `role: "system"`, which gets hoisted into the - provider's system block and busts the system-prompt cache breakpoint - (measured: 7 distinct instruction hashes across one run vs. 1 for the - control). - -Every test class below is named for the specific failure mode it guards -against, treating the donor's design adversarially rather than assuming its -ideas are safe just because the PROMPT is good. - -Coverage: - - the strategy fires (async, early) and absorbs on swap-in - - absorbed messages are sticky-recorded -- byte-identical re-serialization - across repeated calls, and prefix-stable (append-only) as history grows - - tool_calls/tool_result pairs are NEVER split at the absorb boundary - (the donor's exact production failure), at both the unit - (_snap_absorb_boundary) and integration (full swap) level - - the summary message is role="user", enveloped in - , and NOT ephemeral - - fallback to progressive compaction on summarizer failure/timeout/absent - provider -- a turn is never blocked and never fails - - config validation never crashes on a bad compaction_strategy value - - compaction_strategy="summary" composes with token_meter in both modes - - default ("progressive") mode remains completely untouched -- see also - test_default_mode_byte_identical_with_summary_fields_present below, - which is this file's own contribution to the "existing 76 tests green" - guarantee (the full existing suite is run unmodified against this same - source tree as a separate step). -""" - -import asyncio -import logging - -import pytest -from amplifier_core import ChatResponse, TextBlock -from amplifier_module_context_simple import SimpleContextManager, mount - - -class _FakeProvider: - """Minimal stand-in for a Provider -- just enough of `.complete()` to - drive the summarizer, with knobs for failure/timeout/echo-back testing. - Deliberately has neither `get_model_info` nor `get_info`, so - `_calculate_budget` falls back to `self.max_tokens` (matching how the - rest of this module's test suite avoids needing a real provider).""" - - def __init__(self, response_text: str = "SUMMARY TEXT", delay: float = 0.0, raise_exc: Exception | None = None): - self.response_text = response_text - self.delay = delay - self.raise_exc = raise_exc - self.calls: list = [] - - async def complete(self, request): - self.calls.append(request) - if self.delay: - await asyncio.sleep(self.delay) - if self.raise_exc is not None: - raise self.raise_exc - return ChatResponse(content=[TextBlock(type="text", text=self.response_text)]) - - -class _FakeHooks: - """Minimal stand-in for amplifier_core.hooks.HookRegistry -- records - every emitted event so tests can assert on the summarization lifecycle - without depending on real HookRegistry internals.""" - - def __init__(self): - self.emitted: list[tuple[str, dict]] = [] - - async def emit(self, event, data): - self.emitted.append((event, data)) - - -def _tool_call(call_id: str, tool: str = "bash") -> dict: - return { - "role": "assistant", - "content": "", - "tool_calls": [{"id": call_id, "tool": tool, "arguments": {}}], - } - - -def _tool_result(call_id: str, content: str = "result") -> dict: - return {"role": "tool", "tool_call_id": call_id, "content": content} - - -async def _await_pending_task(context: SimpleContextManager) -> None: - """Wait for an in-flight background summarization task to finish, - grabbing the reference before it gets cleared in the task's own - `finally` block.""" - task = context._summarization_task - assert task is not None, "expected a background summarization task to be in flight" - await task - - -def _strip_timestamps(messages: list[dict]) -> list[dict]: - """Normalize out add_message()'s wall-clock timestamp so byte-stability - comparisons focus on content/structure, not incidental timing.""" - result = [] - for msg in messages: - meta = dict(msg.get("metadata") or {}) - meta.pop("timestamp", None) - result.append({**msg, "metadata": meta}) - return result - - -# --------------------------------------------------------------------------- -# Config validation: never crash on a bad compaction_strategy value -# --------------------------------------------------------------------------- - - -def test_default_compaction_strategy_is_progressive(): - context = SimpleContextManager() - assert context.compaction_strategy == "progressive" - - -def test_invalid_compaction_strategy_falls_back_to_progressive_with_warning(caplog): - with caplog.at_level(logging.WARNING): - context = SimpleContextManager(compaction_strategy="bogus") - - assert context.compaction_strategy == "progressive" - assert any("unknown compaction_strategy" in rec.message for rec in caplog.records) - - -@pytest.mark.asyncio -async def test_mount_invalid_compaction_strategy_falls_back_with_warning(caplog): - class _Coordinator: - def __init__(self): - self.hooks = None - self.mounted = {} - - async def mount(self, kind, instance): - self.mounted[kind] = instance - - coordinator = _Coordinator() - with caplog.at_level(logging.WARNING): - await mount(coordinator, {"compaction_strategy": "bogus"}) - - assert coordinator.mounted["context"].compaction_strategy == "progressive" - assert any("unknown compaction_strategy" in rec.message for rec in caplog.records) - - -@pytest.mark.asyncio -async def test_mount_threads_summary_config_through(): - class _Coordinator: - def __init__(self): - self.hooks = None - self.mounted = {} - - async def mount(self, kind, instance): - self.mounted[kind] = instance - - coordinator = _Coordinator() - await mount( - coordinator, - { - "compaction_strategy": "summary", - "summary_trigger": 0.45, - "summarization_model": "gpt-test", - "summarization_timeout_s": 5.0, - }, - ) - context = coordinator.mounted["context"] - assert context.compaction_strategy == "summary" - assert context.summary_trigger == 0.45 - assert context.summarization_model == "gpt-test" - assert context.summarization_timeout_s == 5.0 - - -# --------------------------------------------------------------------------- -# Default mode ("progressive") is completely untouched -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_default_mode_byte_identical_with_summary_fields_present(): - """A manager with compaction_strategy left at its default must produce - byte-identical output to one explicitly asking for "progressive" -- - the new fields/branches must be true no-ops, never just usually-empty.""" - baseline = SimpleContextManager( - max_tokens=1_000, compact_threshold=0.5, compaction_notice_enabled=False - ) - explicit = SimpleContextManager( - max_tokens=1_000, - compact_threshold=0.5, - compaction_notice_enabled=False, - compaction_strategy="progressive", - ) - for i in range(20): - msg = {"role": "user", "content": f"message {i} " + "x" * 50} - await baseline.add_message(dict(msg)) - await explicit.add_message(dict(msg)) - - baseline_view = await baseline.get_messages_for_request() - explicit_view = await explicit.get_messages_for_request() - - assert _strip_timestamps(baseline_view) == _strip_timestamps(explicit_view) - # Never even glances at a provider or spawns a task in default mode. - assert baseline._cached_provider is None - assert baseline._summarization_task is None - assert baseline._pending_summary is None - - -@pytest.mark.asyncio -async def test_progressive_mode_never_triggers_summarizer_even_with_provider(): - """Passing a provider to get_messages_for_request() in the default - ("progressive") mode must never cache it or touch any summary state -- - those branches are gated on compaction_strategy == "summary" only.""" - context = SimpleContextManager( - max_tokens=200, compact_threshold=0.5, compaction_notice_enabled=False - ) - provider = _FakeProvider() - for i in range(20): - await context.add_message({"role": "user", "content": f"msg {i} " + "x" * 30}) - - await context.get_messages_for_request(provider=provider) - - assert context._cached_provider is None - assert context._is_summarizing is False - assert provider.calls == [] - - -# --------------------------------------------------------------------------- -# Strategy fires (early, async) and absorbs on swap-in -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_summary_strategy_fires_absorbs_and_swaps_in(): - context = SimpleContextManager( - compaction_strategy="summary", - compaction_notice_enabled=False, - protected_recent=0.5, - summary_trigger=0.3, - target_usage=0.2, - compact_threshold=0.99, # keep the outer progressive gate CLOSED for now - max_tokens=1_000_000, # corrected below once real usage is known - ) - for i in range(30): - await context.add_message({"role": "user", "content": f"user turn {i} " + "x" * 40}) - await context.add_message( - {"role": "assistant", "content": f"assistant reply {i} " + "y" * 40} - ) - - raw_tokens = context._estimate_tokens(context.messages) - # Usage sits comfortably above summary_trigger (0.3) but below - # compact_threshold (0.99): the summary trigger should fire; the outer - # progressive gate should not. - context.max_tokens = int(raw_tokens / 0.5) - - provider = _FakeProvider(response_text="COMPACT SUMMARY OF EARLY TURNS") - view1 = await context.get_messages_for_request(provider=provider) - - assert context._is_summarizing is True - assert context._pending_summary is None, "must not resolve synchronously" - assert context._last_compaction_stats is None, "outer gate must stay closed" - assert view1 is not None - - await _await_pending_task(context) - - assert len(provider.calls) == 1 - assert context._pending_summary is not None - assert context._is_summarizing is False - assert context._removed_seqs == set(), "must not absorb until the outer gate actually fires" - - # Now open the outer gate so the pending summary gets swapped in. Set - # comfortably ABOVE target_usage (0.2) so the post-absorption level no - # longer "exceeds threshold" and no progressive level is also needed -- - # but still below the pre-swap ~0.5 usage, so the gate actually opens. - context.compact_threshold = 0.3 - view2 = await context.get_messages_for_request(provider=provider) - - assert context._last_compaction_stats is not None - assert context._last_compaction_stats["strategy_level"] == 0, ( - "summary alone should resolve this pass with no progressive level needed" - ) - assert context._removed_seqs, "absorbed messages must be recorded removed" - - summary_msgs = [ - m - for m in view2 - if (m.get("metadata") or {}).get("type") == "context_summary" - ] - assert len(summary_msgs) == 1 - assert "COMPACT SUMMARY OF EARLY TURNS" in summary_msgs[0]["content"] - - # The absorbed originals must be gone from the served view. - assert not any( - isinstance(m.get("content"), str) and "user turn 0 " in m["content"] for m in view2 - ) - - -@pytest.mark.asyncio -async def test_summary_never_re_absorbs_its_own_past_summary_message(): - """A prior summary message (metadata.type == "context_summary") must - never itself become a candidate for a later absorption round -- this - PR deliberately does not implement tier merging (see module docstring); - each escalation produces its own standalone summary.""" - context = SimpleContextManager(compaction_strategy="summary", protected_recent=0.1) - await context.add_message({"role": "user", "content": "hello"}) - summary_msg = context._make_summary_message("a past summary") - context.messages.append(summary_msg) - for i in range(10): - await context.add_message({"role": "user", "content": f"turn {i} " + "z" * 30}) - - seqs = context._select_summary_absorb_seqs(excess_tokens=10_000) - summary_seq = summary_msg["metadata"]["_seq"] - assert seqs is None or summary_seq not in seqs - - -# --------------------------------------------------------------------------- -# Sticky recording + byte/prefix stability across repeated calls -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_absorbed_span_is_byte_identical_across_repeated_calls(): - """Once absorbed, repeated get_messages_for_request() calls (with no - new pending summary) must reproduce the exact same view -- this is what - _record_removed + _apply_sticky_decisions guarantee by construction.""" - context = SimpleContextManager( - compaction_strategy="summary", - compaction_notice_enabled=False, - protected_recent=0.5, - summary_trigger=0.1, - target_usage=0.05, - compact_threshold=0.05, - max_tokens=1_000_000, - ) - for i in range(20): - await context.add_message({"role": "user", "content": f"turn {i} " + "x" * 40}) - await context.add_message({"role": "assistant", "content": f"reply {i} " + "y" * 40}) - - raw_tokens = context._estimate_tokens(context.messages) - context.max_tokens = int(raw_tokens / 0.9) # comfortably above every threshold above - - provider = _FakeProvider(response_text="STABLE SUMMARY") - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - view_a = await context.get_messages_for_request(provider=provider) - view_b = await context.get_messages_for_request(provider=provider) - - assert _strip_timestamps(view_a) == _strip_timestamps(view_b) - - -@pytest.mark.asyncio -async def test_prefix_is_append_only_as_new_turns_arrive_after_a_swap(): - """After a summary swap, growing the conversation further must only - ever APPEND to the previously-served view, never reorder or rewrite - the shared prefix -- the property prompt caching depends on.""" - context = SimpleContextManager( - compaction_strategy="summary", - compaction_notice_enabled=False, - protected_recent=0.5, - summary_trigger=0.1, - target_usage=0.05, - compact_threshold=0.99, # closed until the summarizer has had time to finish - max_tokens=1_000_000, - ) - for i in range(20): - await context.add_message({"role": "user", "content": f"turn {i} " + "x" * 40}) - await context.add_message({"role": "assistant", "content": f"reply {i} " + "y" * 40}) - - raw_tokens = context._estimate_tokens(context.messages) - context.max_tokens = int(raw_tokens / 0.9) - - provider = _FakeProvider(response_text="STABLE SUMMARY") - await context.get_messages_for_request(provider=provider) # fires the trigger only - await _await_pending_task(context) - - # Open the outer gate (target_usage=0.05 leaves ample headroom under it, - # so growing by one small message afterward will NOT re-open escalation). - context.compact_threshold = 0.3 - view_before = _strip_timestamps(await context.get_messages_for_request(provider=provider)) - assert context._last_compaction_stats is not None # the swap actually ran - - await context.add_message({"role": "user", "content": "one more turn"}) - view_after = _strip_timestamps(await context.get_messages_for_request(provider=provider)) - - assert view_after[: len(view_before)] == view_before - assert view_after[len(view_before) :] == [ - {"role": "user", "content": "one more turn", "metadata": {}} - ] - - -# --------------------------------------------------------------------------- -# Tool-pair atomicity at the absorb boundary -- the donor's exact failure -# --------------------------------------------------------------------------- - - -class TestSnapAbsorbBoundaryUnit: - """Direct unit tests for _snap_absorb_boundary -- the fix for the - donor's `_snap_to_tool_pair_boundary`, which only checked adjacency and - did no protected-boundary accounting, and in production dropped a - `function_call` while keeping its `function_call_output`.""" - - def test_extends_to_include_a_straddling_result(self): - context = SimpleContextManager(compaction_strategy="summary") - live = [ - {"role": "user", "content": "u0"}, - _tool_call("call_1"), - _tool_result("call_1"), - {"role": "user", "content": "u1"}, - ] - # end_idx=2 would include the call but exclude its own result. - assert context._snap_absorb_boundary(live, 2, protected_boundary=4) == 3 - - def test_extends_to_include_a_non_adjacent_straggler_result(self): - """The donor's adjacency-only heuristic misses this: the result is - not the message immediately following the call.""" - context = SimpleContextManager(compaction_strategy="summary") - live = [ - _tool_call("call_1"), - {"role": "assistant", "content": "unrelated narration"}, - _tool_result("call_1"), - ] - assert context._snap_absorb_boundary(live, 1, protected_boundary=3) == 3 - - def test_shrinks_to_exclude_a_call_whose_result_is_protected(self): - """When extending would cross into the protected tail, the whole - pair must be excluded -- never split, never absorbed partially.""" - context = SimpleContextManager(compaction_strategy="summary") - live = [ - {"role": "user", "content": "u0"}, - _tool_call("call_1"), - _tool_result("call_1"), - {"role": "user", "content": "u1"}, - ] - # protected_boundary=2: the result at index 2 is already protected. - assert context._snap_absorb_boundary(live, 2, protected_boundary=2) == 1 - - def test_multiple_results_one_straddling_excludes_the_whole_call(self): - context = SimpleContextManager(compaction_strategy="summary") - live = [ - { - "role": "assistant", - "content": "", - "tool_calls": [{"id": "a", "tool": "x", "arguments": {}}, {"id": "b", "tool": "y", "arguments": {}}], - }, - _tool_result("a"), - _tool_result("b"), - ] - # protected_boundary=2 protects the second result (index 2) -> the - # whole call (and its first, otherwise-includable result) must go. - assert context._snap_absorb_boundary(live, 2, protected_boundary=2) == 0 - - def test_clean_pair_within_bounds_is_unchanged(self): - context = SimpleContextManager(compaction_strategy="summary") - live = [_tool_call("call_1"), _tool_result("call_1"), {"role": "user", "content": "u1"}] - assert context._snap_absorb_boundary(live, 2, protected_boundary=3) == 2 - - def test_zero_boundary_returns_zero(self): - context = SimpleContextManager(compaction_strategy="summary") - assert context._snap_absorb_boundary([], 0, protected_boundary=0) == 0 - - -@pytest.mark.asyncio -async def test_integration_tool_pair_never_split_at_absorb_boundary(): - """End-to-end: a tool_calls/tool_result pair sitting right at the - natural absorb boundary must be absorbed (or not) as an atomic unit -- - never left with one half served and the other half gone, which is - exactly the InvalidRequestError the donor shipped in production.""" - context = SimpleContextManager( - compaction_strategy="summary", - compaction_notice_enabled=False, - protected_recent=0.3, - summary_trigger=0.1, - target_usage=0.05, - compact_threshold=0.05, - max_tokens=1_000_000, - ) - for i in range(10): - await context.add_message({"role": "user", "content": f"turn {i} " + "x" * 60}) - await context.add_message({"role": "assistant", "content": f"reply {i} " + "y" * 60}) - # A tool pair positioned squarely in what will become the absorb - # candidate pool (well before the protected tail). - await context.add_message(_tool_call("straddle_call")) - await context.add_message(_tool_result("straddle_call", "tool output payload")) - for i in range(10, 20): - await context.add_message({"role": "user", "content": f"turn {i} " + "x" * 60}) - await context.add_message({"role": "assistant", "content": f"reply {i} " + "y" * 60}) - - raw_tokens = context._estimate_tokens(context.messages) - context.max_tokens = int(raw_tokens / 0.9) - - provider = _FakeProvider(response_text="SUMMARY") - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - view = await context.get_messages_for_request(provider=provider) - - has_call = any(m.get("role") == "assistant" and m.get("tool_calls") for m in view) - has_result = any(m.get("role") == "tool" for m in view) - assert has_call == has_result, ( - f"tool pair split at the absorb boundary! has_call={has_call} has_result={has_result}" - ) - if has_call: - for i, m in enumerate(view): - if m.get("role") == "assistant" and m.get("tool_calls"): - assert i + 1 < len(view) and view[i + 1].get("role") == "tool" - - -# --------------------------------------------------------------------------- -# Summary message shape: role=user, enveloped, NOT ephemeral -# --------------------------------------------------------------------------- - - -def test_summary_message_is_user_role_enveloped_and_non_ephemeral(): - context = SimpleContextManager(compaction_strategy="summary") - msg = context._make_summary_message("the summary body") - - assert msg["role"] == "user", "must never be role=system -- see module docstring" - assert msg["content"].startswith('') - assert msg["content"].endswith("") - assert "the summary body" in msg["content"] - assert msg["metadata"]["type"] == "context_summary" - assert "ephemeral" not in msg["metadata"], ( - "must NOT be marked ephemeral -- it is meant to persist as stable history" - ) - assert "_seq" in msg["metadata"] - - -def test_summary_message_gets_a_fresh_seq_like_add_message_would(): - context = SimpleContextManager(compaction_strategy="summary") - before = context._next_seq - msg = context._make_summary_message("text") - assert msg["metadata"]["_seq"] == before - assert context._next_seq == before + 1 - - -# --------------------------------------------------------------------------- -# Fallback to progressive compaction: failure / timeout / no provider -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_fallback_to_progressive_on_summarizer_exception(): - context = SimpleContextManager( - compaction_strategy="summary", - compaction_notice_enabled=False, - protected_recent=0.3, - summary_trigger=0.1, - target_usage=0.2, - compact_threshold=0.1, - max_tokens=1_000_000, - ) - for i in range(20): - await context.add_message({"role": "user", "content": f"turn {i} " + "x" * 40}) - await context.add_message({"role": "assistant", "content": f"reply {i} " + "y" * 40}) - - raw_tokens = context._estimate_tokens(context.messages) - context.max_tokens = int(raw_tokens / 0.9) - - provider = _FakeProvider(raise_exc=RuntimeError("summarizer is down")) - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - assert context._pending_summary is None - assert context._summarization_failures == 1 - - # A second call must still make forward progress via the progressive - # ladder -- never blocked, never raised, even though summarization just - # failed. - view2 = await context.get_messages_for_request(provider=provider) - assert context._last_compaction_stats is not None - assert context._last_compaction_stats["strategy_level"] >= 1 - assert len(view2) <= len(context.messages) - - -@pytest.mark.asyncio -async def test_fallback_to_progressive_on_summarizer_timeout(): - context = SimpleContextManager( - compaction_strategy="summary", - compaction_notice_enabled=False, - protected_recent=0.3, - summary_trigger=0.1, - target_usage=0.2, - compact_threshold=0.1, - max_tokens=1_000_000, - summarization_timeout_s=0.01, - ) - for i in range(20): - await context.add_message({"role": "user", "content": f"turn {i} " + "x" * 40}) - await context.add_message({"role": "assistant", "content": f"reply {i} " + "y" * 40}) - - raw_tokens = context._estimate_tokens(context.messages) - context.max_tokens = int(raw_tokens / 0.9) - - provider = _FakeProvider(delay=1.0) # far longer than summarization_timeout_s - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - assert context._pending_summary is None - assert context._summarization_failures == 1 - - view2 = await context.get_messages_for_request(provider=provider) - assert context._last_compaction_stats is not None - assert context._last_compaction_stats["strategy_level"] >= 1 - assert len(view2) <= len(context.messages) - - -@pytest.mark.asyncio -async def test_fallback_to_progressive_when_no_provider_ever_passed(): - """compaction_strategy="summary" but the caller never passes a - provider (e.g. a code path that doesn't support it yet) -- must behave - exactly like progressive compaction, never raise, never hang.""" - context = SimpleContextManager( - compaction_strategy="summary", - max_tokens=100, - compact_threshold=0.5, - compaction_notice_enabled=False, - ) - for i in range(20): - await context.add_message({"role": "user", "content": f"msg {i}"}) - await context.add_message({"role": "assistant", "content": f"reply {i} " + "z" * 20}) - - view = await context.get_messages_for_request() # provider=None (default) - - assert context._cached_provider is None - assert context._is_summarizing is False - assert context._last_compaction_stats is not None - assert "messages_absorbed_by_summary" in context._last_compaction_stats - assert context._last_compaction_stats["messages_absorbed_by_summary"] == 0 - assert len(view) < len(context.messages) - - -@pytest.mark.asyncio -async def test_never_triggers_twice_while_one_is_already_in_flight(): - context = SimpleContextManager( - compaction_strategy="summary", - compaction_notice_enabled=False, - protected_recent=0.5, - summary_trigger=0.1, - target_usage=0.05, - compact_threshold=0.99, - max_tokens=1_000_000, - ) - for i in range(20): - await context.add_message({"role": "user", "content": f"turn {i} " + "x" * 40}) - await context.add_message({"role": "assistant", "content": f"reply {i} " + "y" * 40}) - raw_tokens = context._estimate_tokens(context.messages) - context.max_tokens = int(raw_tokens / 0.5) - - provider = _FakeProvider(delay=0.05) - await context.get_messages_for_request(provider=provider) - assert context._is_summarizing is True - await asyncio.sleep(0) # let the task actually start running (still in-flight, delay=0.05) - await context.get_messages_for_request(provider=provider) - await context.get_messages_for_request(provider=provider) - assert context._is_summarizing is True, "still in flight -- the delay hasn't elapsed yet" - - await _await_pending_task(context) - assert len(provider.calls) == 1, "must not fire a second concurrent summarization call" - - -@pytest.mark.asyncio -async def test_stale_pending_summary_discarded_gracefully_not_a_failure(): - """If the absorbed span was already fully resolved by an intervening - escalation before the swap runs, the pending summary must be discarded - quietly -- NOT counted as a failure, and NOT crash.""" - context = SimpleContextManager(compaction_strategy="summary", protected_recent=0.9) - for i in range(5): - await context.add_message({"role": "user", "content": f"turn {i}"}) - - seqs = [m["metadata"]["_seq"] for m in context.messages[:2]] - context._pending_summary = {"seqs": frozenset(seqs), "text": "irrelevant"} - # Simulate an intervening escalation that already removed these seqs. - for s in seqs: - context._removed_seqs.add(s) - - non_system = [m for m in context.messages if m.get("role") != "system"] - result, did_swap = await context._swap_in_pending_summary(non_system) - - assert did_swap is False - assert result == non_system - assert context._pending_summary is None - assert context._summarization_failures == 0 - - -# --------------------------------------------------------------------------- -# Composes with the real-usage token meter, both modes -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_summary_strategy_works_with_token_meter_estimate_mode(): - """token_meter left at its default ("estimate"): the summary trigger - must be driven by the same estimator the outer gate uses -- no crash, - no divergence in which signal feeds which check.""" - context = SimpleContextManager( - compaction_strategy="summary", - token_meter="estimate", - compaction_notice_enabled=False, - protected_recent=0.5, - summary_trigger=0.3, - target_usage=0.2, - compact_threshold=0.99, - max_tokens=1_000_000, - ) - for i in range(20): - await context.add_message({"role": "user", "content": f"turn {i} " + "x" * 40}) - await context.add_message({"role": "assistant", "content": f"reply {i} " + "y" * 40}) - raw_tokens = context._estimate_tokens(context.messages) - context.max_tokens = int(raw_tokens / 0.5) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - - assert context._last_token_meter_stats["mode"] == "estimate" - assert context._is_summarizing is True - await _await_pending_task(context) - - -@pytest.mark.asyncio -async def test_summary_strategy_works_with_token_meter_actual_mode(): - """token_meter="actual": a real llm:response measurement, once - observed, must be able to drive the EARLY summary trigger too (not just - the outer progressive gate) -- _maybe_trigger_summary_compaction reuses - the same already-meter-aware `token_count`.""" - context = SimpleContextManager( - compaction_strategy="summary", - token_meter="actual", - compaction_notice_enabled=False, - protected_recent=0.5, - summary_trigger=0.3, - target_usage=0.2, - compact_threshold=0.99, - max_tokens=100_000, - ) - for i in range(6): - await context.add_message({"role": "user", "content": f"msg {i}"}) - await context.add_message({"role": "assistant", "content": f"reply {i}"}) - - # Estimator alone would NOT cross summary_trigger (0.3) -- real usage - # must be what drives the trigger here. - estimate = context._estimate_tokens(context.messages) - assert estimate / 100_000 < 0.3 - - await context._on_llm_response("llm:response", {"usage": {"input_tokens": 40_000}}) - - provider = _FakeProvider() - await context.get_messages_for_request(provider=provider) - - assert context._last_token_meter_stats["source"] == "measured" - assert context._is_summarizing is True, ( - "the real measurement (40_000/100_000 = 0.4) should have crossed " - "summary_trigger (0.3) even though the estimator alone would not" - ) - await _await_pending_task(context) - - -# --------------------------------------------------------------------------- -# Observability: hook events fire around the summarization lifecycle -# --------------------------------------------------------------------------- - - -@pytest.mark.asyncio -async def test_hooks_receive_pre_and_post_summarize_events(): - hooks = _FakeHooks() - context = SimpleContextManager( - compaction_strategy="summary", - compaction_notice_enabled=False, - protected_recent=0.5, - summary_trigger=0.3, - target_usage=0.2, - compact_threshold=0.99, - max_tokens=1_000_000, - hooks=hooks, - ) - for i in range(20): - await context.add_message({"role": "user", "content": f"turn {i} " + "x" * 40}) - await context.add_message({"role": "assistant", "content": f"reply {i} " + "y" * 40}) - raw_tokens = context._estimate_tokens(context.messages) - context.max_tokens = int(raw_tokens / 0.5) - - provider = _FakeProvider(response_text="SUMMARY") - await context.get_messages_for_request(provider=provider) - await _await_pending_task(context) - - events = [name for name, _ in hooks.emitted] - assert "context:pre_summarize" in events - assert "context:post_summarize" in events