Skip to content

extract_llm_sweep: the inventory's "tool_use id" is a content hash, so the locating anchor anchors nothing #123

Description

@amiddavid

Found by live verification of #118 with a verbatim wire recorder.

The inventory field presented to the model as a tool_use id is not one. extract_sweep.go:266 sets id := extract.ContentKey(content) — a 24-hex sha256 prefix, which is also the result-cache and stash key — and :430 passes it as AdjudicationItem.ID. adjudicate.go:173 renders it as ", tool_use id ", and the contract tells the model "The tool_use id is shown only so you can find the output in the conversation above."

On the wire:

[0] 12947 tokens, tool_use id 300c312d1492952219bfb1c4, begins: # ledger_b

The real id for that output in that transcript is toolu_d2. The string shipped appears nowhere in the conversation. So the field does not merely fail to help — it tells the model to locate content by a key that cannot be found, which is worse than omitting it.

This is the head commit's own headline claim. 4a67916 is titled "ship the tool_use id as a locating anchor", and it is a change I specifically asked for: the reasoning was that the id is the one exact anchor between an inventory entry and the content the model reads from cache, since head-plus-size is weak on a transcript of near-identical Read results. That reasoning was right; the implementation ships a different value.

The test cannot see it. adjudicate_test.go:235 hard-codes ID: "toolu_abc123" and asserts only that BuildPrefixAsk renders whatever is in the field. It verifies the plumbing, never the provenance — so it passes on any string, including one that appears in no transcript.

The fix, and why it was not applied

The real id is one field away: normalize sets ChatToolMessage.ToolCallID on every synthetic tool message (apply/apply.go:1335). But it needs a second field on sweepCand — the existing id is load-bearing as the store/stash key and must not change — and it alters prompt content on a component whose prompt effects are measured, so the yield call belongs to the author rather than to a verification pass.

A test for it should assert provenance, not rendering: that the id shipped for a candidate is the tool_use_id present in the request that produced it. Anything weaker passes on the current bug.

Interaction with #122

Worth sequencing after #122. If the candidate set changes so the model actually reads the outputs it judges, the anchor's job changes with it — under #122's option (2) the candidates live inside the cached region and an exact id becomes genuinely load-bearing rather than nice to have.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions