extract_llm_sweep: empty and wrong-tool prefix-ask replies are both filed as sweep_unparseable

[amiddavid]

What

The extract_llm_sweep prefix ask sometimes returns a reply the component reads as the empty
string, which lands in sweep_unparseable — a counter whose name says "the prompt is wrong". It is
not a prompt problem, and it is the dominant residual failure mode in every arm of the
three-way A/B run for #137.

How it was found

Measuring #137. Every unusable reply is logged (cg.sweep.unusable_reply), and splitting them by
shape across three benchmark passes per arm (aws/claude-sonnet-5, LOCA S2L cg_128k_3):

arm	unusable	of which EMPTY (reply_len == 0)	of which prose	of which JSONL
origin/main (tool_choice: none)	6	5	1	0
tool_choice removed, no verdict tool	14	9 (sampled)	0	1
#137 as merged-to-be	7	5	0	0

Note the sample log is capped at maxSweepUnusableSamples, so those columns undercount the larger
arm.

A dedicated pass with every reply's content blocks logged shows what the empty replies actually
are. 29 prefix-ask replies, tool_choice removed and no verdict tool declared:

thinking,text                                        20
thinking,tool_use:context_guru_expand                 5
thinking,tool_use:canvas_canvas_list_announcements     2
thinking,text,tool_use:context_guru_expand            1
thinking,tool_use:memory_read_graph                   1

8 of 29 replies (27.6%) carry a tool_use and no text block at all.
cheapmodel.CompletePrefixed returns the first non-empty text, so those read as "".

Two distinct causes hide behind one counter:

1. The model called somebody else's tool. The ask appends to a prefix carrying the agent's own
   tools (they are in the cache key, so they cannot be stripped) plus context_guru_expand. With
   nothing better on offer the model calls one of those. fix(adjudicate): declare a verdict tool and stop forcing tool_choice:none #137 removes most of this by declaring a
   tool worth calling — 55.8% of its replies come back through it — but it cannot remove all of it.
2. A genuine thinking-only reply, no text and no tool call. Present on main too, where
   tool_choice: none makes case 1 impossible, so main's 5 empties are all of this kind.

Severity

Low-to-moderate, and mostly a diagnosis defect. Everything fails open: an unusable reply keeps
every output verbatim, so no content is lost. The cost is a wasted ask (real tokens, and up to 90 s
of llmCallTimeout risk) plus a counter that actively misdirects: an operator reading
sweep_unparseable will go and edit the prompt, when for case 1 the prompt was never read.

Fix options

1. Split the counter (small, and worth doing regardless). sweep_reply_empty for
   reply_len == 0, and sweep_reply_other_tool when the reply's only tool_use names a tool that
   is not ours. Today sweep_unparseable conflates three failures needing three different fixes.
   Cheap, no behaviour change; the reply shape is already parsed in CompletePrefixed.
2. Read the thinking block as a last resort for case 2. Tempting and probably wrong: a verdict
   recovered from reasoning the model did not commit to is exactly the input
   sweep_quote_fabricated exists to catch.
3. Retry once on an empty reply. Bounded, but it doubles the worst-case latency on a path that
   already sits inside a 90 s timeout, and it pays a second prefix read. Only defensible once the
   counter split shows how often case 2 actually happens.
4. Do nothing about the behaviour, fix only the label. Given fail-open and the low rate after
   fix(adjudicate): declare a verdict tool and stop forcing tool_choice:none #137 (7 of 77 asks), this is a reasonable resting point — but option 1 is the prerequisite for
   ever knowing that.

Recommendation: option 1 now, then re-measure before considering 3.

Found while measuring #137; not folded into it because it is present in origin/main at the same
rate and is not caused or cured by that change.