feat: span-size predicate in front of the summarizer fork (summary_fork_min_span_ratio / summary_call_mode: auto) - #29
Conversation
Lane 6da measured summary_call_mode="fork" end-to-end and found it
cost-neutral: the cache mechanism works (a forked call reads a median
85.7% of its own prompt from cache vs 0.0% standalone) but total run
cost moved -0.8%, i.e. noise. It cancels because a forked call is 4.5x
cheaper per token ($0.00077 vs $0.00350 per 1k own-prompt tokens) while
its prompt is 5.2x bigger (median 26,620 vs 5,129 tokens).
The trade is span-size dependent, and that is the lever. A standalone
call pays for the SPAN at the uncached rate; a fork pays for the whole
PREFIX at the cached rate. So the fork is cheaper exactly when
P x 0.00077 < S x 0.00350 <=> S / P > 0.22
0.22 is DEFAULT_FORK_MIN_SPAN_RATIO -- derived from those two measured
rates, not chosen. It classifies both of 6da's measured buckets the way
the money went (spans <=15k: standalone $0.0151 < fork $0.0265, declines;
spans >30k: standalone $0.1410 vs flat ~$0.027, forks), and it sits just
above 6da's median ratio of 0.193, which is exactly why 6da measured the
two arms cancelling at the median.
Adds:
- summary_fork_min_span_ratio (default None = predicate off, not even
evaluated)
- summary_call_mode "auto" = fork gated at the measured default
- fork_declines + span_measure in last_summary_call_stats
TWO defaults are held still, not one: "standalone" is byte-identical as
before, AND plain "fork" without an explicit ratio still forks
unconditionally exactly as PR #27 shipped it.
A decline is not a fallback. The predicate runs strictly AFTER every
alignment precondition, so it only ever sees forks that would have
worked and declines them on cost alone; a misaligned fork can never be
bought by making the span bigger. fork_declines and fork_fallbacks stay
separate counters because one combined number would make a healthy
predicate look identical to a broken fork.
This does NOT lift 6da's structural 45.5% fork-rate cap -- running after
the refusal checks, it can only lower the realized fork rate, never
raise it. See probes/pmt-fork-span-predicate/DONE-NOTE.md section 5.
Tests: 31 new (tests/test_summary_fork_span_predicate.py), boundary
pinned from both sides against the fixture's own realized ratio, and
mutation-checked -- 4 deliberate mutations each caught by 2-10 tests.
Full suite 312 passed, 1 skipped (baseline 281/1). ruff clean.
Evidence: .amplifier/evaluation/probes/6da-summary-fork/FINDINGS.md
section 6, in the openai-evals-team-ci repo.
Merge-queue verification — BLOCKED, not merging (real conflict, not independent)Fresh scratch clone at What #30 did, and why it matters here#30 is a deliberate, owner-approved policy revert: "main carries wins only... Unproven default-off feature -- belongs on a branch for evaluation, not on main." Among the six features it reverted is #27 PR #29 is a predicate gating #27's fork mode. It does not stand alone — every line of its diff ( #30's own PR body says this explicitly, naming PR #29 by number:
Verification performed
Why I am not merging, and not attempting a synthetic resolutionThis is exactly the case the task instructions distinguish from a normal conflict: the two sides are not independent hunks that both deserve to survive. #30 is a deliberate, same-day, owner-approved policy decision to remove the unproven feature #29 extends. Forcing a "union" merge here would mean overriding that policy decision to land a PR whose entire premise (a predicate in front of a fork mode) requires the fork mode to exist — silently reintroducing exactly what #30 just removed, under cover of "resolving a conflict." That would be papering over it, not resolving it. The correct next step (already named in #30's own body) is for lane Leaving PR #29 as draft, not ready-for-review, not merged. No gate table gets a clean pass/fail here because the premise gate — "does this apply to current |
1 participant
Gate
summary_call_modefork behind a span-size predicate — fork only when the span is large relative to the prefix. Default no-op in both fork modes.Work item:
model_performance-pmt(discovered frommodel_performance-6da). Base:main@a877b36, the PR #27 merge commit 6da measured. Spend: $0 — no eval was run; 6da's measurement is cited by file throughout.DONE-NOTE —
model_performance-pmtSubject: context-simple — gate
summary_call_modefork behind a span-sizepredicate (fork only when the span is large relative to the prefix).
Branch:
lane/pmt-fork-span-predicate· Base:main@a877b36(the PR #27merge commit lane 6da measured) · Spend: $0.00. No API calls, no DTU, no
eval runs, no infrastructure created. 6da's measurement already exists and is
cited by file throughout.
1. What shipped
Three additions, all default no-op:
summary_fork_min_span_ratioNonespan_tokens / prefix_tokensat which a fork is worth issuing.Noneor0= predicate not evaluated at allsummary_call_mode: "auto""fork"with the predicate on at the measured break-even defaultDEFAULT_FORK_MIN_SPAN_RATIO0.22summary_call_modeis nowstandalone(a.k.a.inline) |fork|auto,matching the item's requested vocabulary. Two defaults, not one, are held
still:
standaloneis byte-identical as before, and plainforkwithoutan explicit ratio still forks unconditionally, exactly as PR #27 shipped it. A
predicate that quietly switched itself on for existing
forkusers would be abehaviour change wearing an opt-in's clothes.
Observability:
last_summary_call_statsgainsfork_declines(session-cumulative) and
span_measure(span_tokens,prefix_tokens,span_ratio,min_span_ratio), both alongside the pre-existingfork_fallbacks.2. DELIVERABLES
lane/pmt-fork-span-predicate, default byte-identical, tests green3. The threshold: derived, not chosen
Source:
.amplifier/evaluation/probes/6da-summary-fork/FINDINGS.md§6("WHY G-FORK-COST FAILED — the two effects cancel, exactly"), in the
openai-evals-team-cirepo.6da's two measured per-token rates:
The cost model those rates imply is asymmetric, and that asymmetry is the whole
finding:
plus a fresh rendering of the span). Its cost scales with S, the span.
the span; the appended instruction is a rounding error. Its cost scales with
P, the prefix.
So the fork is cheaper exactly when
0.22 is that ratio and nothing else. It is a break-even, not a margin:
the point where the two calls cost the same. Raising it buys margin, lowering it
forks speculatively — both are one config value away, and neither is a number I
invented.
Two independent cross-checks against the same table
(a) The median cancellation. 6da's median span is 5,129 tok and its median
fork prompt is 26,620 tok → ratio 0.193, just below break-even. That is
exactly why 6da measured the two arms cancelling "exactly" at the median and
recorded a −0.8% (noise) total run cost. A break-even sitting a hair above the
observed median ratio is what that observation predicts.
(b) The bucket table. 6da's span buckets, and what the predicate says about
each:
Implied mean spans are
mean cost ÷ $0.00350 per 1k— the standalone ratefrom the same table. Both measured buckets are classified the way the money
went. This is pinned as a test
(
test_the_default_classifies_6das_own_measured_populations_correctly), so afuture edit to the constant has to argue with the measurement.
Why a ratio and not an absolute token threshold
The item offered either. A ratio is the correct primitive and an absolute
threshold is only correct at one prefix size: fork cost scales with P,
which grows all session, while standalone cost scales with S. The
break-even is a ratio of two rates, so the predicate must be a ratio. At
6da's median prefix the equivalent absolute threshold is ~5,900 span tokens; by
late session the same ratio is a much larger number. This also matches the
item's own acceptance criteria wording ("a configurable fraction of the
recorded prefix tokens").
4. A decline is not a fallback
The predicate is evaluated strictly after every existing alignment
precondition, never before. That ordering is load-bearing in both directions:
size — it pays for the whole conversation as fresh input);
Consequently the two counters answer different questions and are kept apart:
fork_fallbacksfork_declinesSumming them into one number would make a healthy predicate look exactly like a
broken fork. Declines log at
INFO(once per kind, because every declinemessage carries its own token counts and message-level dedupe would dedupe
nothing); refusals keep their existing
WARNING.5. Does the 45.5% fork-rate cap apply? Yes — and strictly.
6da found 20 of 44 treatment-arm summarizer calls actually forked (45.5%), the
other 24 refused in two families — 12 ×
note_request_sent() has never been called, 8 ×the span … is not present in the recorded prefix. The first isstructural on a CLI workload: every turn is a fresh
amplifier run --resumeprocess, and the summary trigger fires inside the first
get_messages_for_request(), strictly before that process has sent anything.The predicate is subject to that same cap, and cannot relieve it. Because
it runs after the refusal checks, it can only ever decline forks that were
already possible. It lowers the realized fork rate; it can never raise it. The
45.5% is an upper bound on what the predicate has any say over.
And the cap bites precisely where the predicate would have helped most.
6da's own numbers: the treatment arm's fallbacks (n=17) cost $0.08719 mean
against $0.02652 for forked calls — the refusals land on the large spans
that accumulate while a fork is impossible. Those are exactly the
high-ratio calls the predicate is designed to route to a fork, and they never
reach it.
So, stated plainly for the PR: on a CLI workload this predicate's realized win
is bounded by the ≤45.5% forkable subset, and the expensive tail is
disproportionately outside that subset. On a long-lived in-process session
(where
note_request_sent()fires before the first trigger) the cap does notapply and the predicate captures the full tail. Lifting the cap is a change to
the seam — arming
note_request_sent()earlier, or persisting the sent-toolsfact across a resume — not to this predicate, and it would multiply this
predicate's value rather than substitute for it.
6. Tests
New file:
tests/test_summary_fork_span_predicate.py— 31 tests, fourgroups, written against how this can silently go wrong rather than how it is
supposed to work:
standaloneneverevaluates the predicate or writes a predicate attribute;
forkwithout anexplicit ratio records no measurement and forks as before;
autoresolves tothe default; an explicit ratio overrides
auto; bad values disable loudly,0disables silently;mount()threads both knobs.pinned from both sides against the fixture's own realized ratio
(
_measure_realized_ratio(), 0.498 in this fixture), separated by a singleepsilon — a predicate tested only at 0.000001 and 99.0 would pass while being
an order of magnitude wrong. Plus the 6da-population replay from §3(b), the
>=inclusive-boundary semantics, a zero-token prefix (no divide-by-zero, noopinion), and ratios >1.0 as a legal "never fork" setting.
in either direction; a misalignment is still a fallback with the predicate
armed at a ratio nothing could fail, and records no
span_measure(so thedistribution an eval arm plots is not contaminated with calls the threshold
never governed); declines log
INFO, neverWARNING, once per kind.byte-for-byte (digest-compared against a control manager that was never in
fork mode), still produces the summary, consumes no
_seq, touches nohistory, selects the same span with tool-pairs intact, and leaves the next
served view byte-identical.
Full suite: 312 passed, 1 skipped (baseline before this change: 281 passed,
1 skipped).
ruff check: clean.The tests were mutation-checked, not just run
A suite that passes first try proves nothing until you make it fail on purpose.
Four mutations, each reverted after:
if ratio >= threshold→if True)autono longer resolves to the defaultfork_fallbacksinstead offork_declinesDEFAULT_FORK_MIN_SPAN_RATIOchanged 0.22 → 0.9The ordering guarantee (alignment before economics) is covered by
test_a_misalignment_is_still_a_fallback_even_with_the_predicate_on, whichasserts
span_measure is Noneon a misaligned call — moving the predicateearlier populates it and fails the test.
One honesty note on the suite
tests/test_compaction_performance.py::test_compaction_scales_sub_quadraticallyfailed once, during a run executed concurrently with a mutation pass on the
same machine. It is a wall-clock ratio assertion (
large/small < 8) and isload-sensitive by construction. It passed 3/3 in isolation and 3/3 in
back-to-back full-suite runs on an unloaded machine. Pre-existing flake under
load, unrelated to this change — recorded rather than quietly re-run.
7. What is NOT claimed
economic justification is entirely 6da's measurement; what is proven here
is structural — default byte-identity in both modes, correct classification of
6da's own two measured populations, and the counter/ordering separation.
criterion ("summarizer cost share falls measurably vs plain fork mode AND
total run cost does not regress, both terms signed") requires an n≥3/arm
S5-CRAC run that this lane's $0 authority does not fund. That criterion is
NOT-POSSIBLE at this budget and is left open, deliberately and on the
record, rather than substituted with an arithmetic counterfactual dressed up
as a result.
arms differing only in
summary_fork_min_span_ratio(off vs 0.22),per-summarizer-call attribution taken from the module's own
ChatResponse.usage— never positionalllm:request/llm:responsepairing, which is the defect that produced the wrong 2.4%/8.3–10.9% figures
— and
b_constraints/c_post_compactionreported per arm so a cost winbought with a retention loss cannot be reported as a win. The new
span_measurefield exists precisely so that arm can plot the realizedspan:prefix distribution and re-derive its own threshold from its own data
instead of trusting this module's default.
8. Open / recommended next
measured verdict).
--resume, orarm
note_request_sent()before the firstget_messages_for_request().Independent of this predicate and worth more because of it: 6da measured
the capped calls at $0.0872 mean, the most expensive population in the run.
00-what-we-know.md§2h conflict feat: respect provider context_budget_cap for cost-optimal compaction #8 still records 2.4% vs 8.3–10.9%. 6dafiled
PROPOSED-CORRECTIONS.mdbeside its FINDINGS; the real figure is~30%. Not this lane's file to edit (lane rule 2), noted so it is not lost.