Skip to content

llm: fix Anthropic thinking/effort payload to match the real API - #100

Open
sandeep wants to merge 1 commit into
laude-institute:mainfrom
sandeep:llm-anthropic-thinking-fix
Open

llm: fix Anthropic thinking/effort payload to match the real API#100
sandeep wants to merge 1 commit into
laude-institute:mainfrom
sandeep:llm-anthropic-thinking-fix

Conversation

@sandeep

@sandeep sandeep commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #99

What

thinking:{type:"adaptive"} and output_config:{effort:...} are not real
fields in Anthropic's public Messages API. The real API rejects both on
claude-sonnet-4-5, its flagship model. shellm always calls bin/llm
with --thinking --effort high, so monolith wakeup fails on any real
Anthropic Claude model. responder calls bin/llm directly, without
--thinking/--effort, so chat replies keep working and can look like
everything is fine.

How it works

  • thinking.type is now "enabled", with a required budget_tokens
    half of max_tokens, floored at the API's 1024 minimum, leaving the
    other half of the token budget for the visible response.
  • output_config.effort is now gated behind a new
    supports_anthropic_effort() guard, mirroring the existing
    supports_anthropic_thinking()/supports_openai_reasoning()/
    supports_gemini_thinking() guards. It returns false for every known
    model, since none of them accept this field via the real API.
    LLM_ASSUME_THINKING=1 is the same escape hatch already used for the
    other guards, for a future model that does support it.
  • Help text (--thinking, the Anthropic section under "Thinking &
    effort") updated to match.

Tests

New tests/test_llm_anthropic_thinking.sh (9 checks, curl stubbed to
capture the outgoing payload): thinking.type is "enabled" not
"adaptive", budget_tokens is half of max_tokens and floors at 1024,
--effort no longer adds output_config and is reported on stderr,
thinking still sends when --effort is dropped alongside it (the exact
combination shellm always sends), and LLM_ASSUME_THINKING=1 forces
output_config.effort through.

Verified against the real Anthropic API (not just the stub): both flags
together on claude-sonnet-4-5 now succeed end to end, live on a
previously-broken identity — monolith went from 100% rc=1 to a normal
multi-iteration run immediately after.

Full tests/run-all.sh: 37/38 pass; the one failure
(test_thinkers_pending.sh) is unrelated dispatcher-queue timing, and
fails the same way on unmodified main in this environment.
shellcheck -S warning, bash -n, and a bash-3.2 parse check are all
clean.

thinking:{type:"adaptive"} and output_config:{effort:...} both looked
plausible but aren't real fields in Anthropic's public Messages API. Found
live against claude-sonnet-4-5 (the flagship model): the real API rejects
"adaptive" outright ("adaptive thinking is not supported on this model")
and rejects output_config.effort on every model tried ("This model does
not support the effort parameter"). shellm always passes --thinking
--effort together, so every shellm call to a real Claude model failed
100% of the time — chat replies (a separate, simpler direct llm call with
no --thinking) were unaffected, which is why this stayed hidden.

- thinking now sends type:"enabled" with a required budget_tokens (half of
  max_tokens, floored at the API's 1024 minimum) instead of "adaptive".
- effort is now gated behind a capability guard (supports_anthropic_effort,
  mirroring supports_anthropic_thinking) that returns false for every
  known model, since none of them accept it — LLM_ASSUME_THINKING=1 is the
  same escape hatch already used for the thinking guards.

New tests/test_llm_anthropic_thinking.sh covers both fixes plus the escape
hatch, with curl stubbed to capture the actual outgoing payload.

@nickjalbert nickjalbert left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! From review:

The patch breaks Claude Opus 4.7 and new Anthropic models

bin/llm:467 treats every current Claude family alike. bin/llm:521 then sends thinking.type: "enabled" with budget_tokens, while bin/llm:483 drops output_config.effort for every model.

That payload fixes Sonnet 4.5, but Claude 4.7 and later reject manual extended thinking and require adaptive thinking with effort. Headlong defaults to Opus 4.7 and always passes both flags, so normal shellm calls would fail. Anthropic also documents effort support for several current models. Anthropic's API primer describes the model split.

Add an explicit model capability table. At minimum, test Sonnet 4.5 with enabled plus a budget, and Opus 4.7 with adaptive plus effort. Include the supported 4.6 behavior as well.

Small max_tokens values produce an invalid budget.

bin/llm:527 floors the thinking budget at 1024 without ensuring it remains below max_tokens. The new test therefore accepts max_tokens: 100 with budget_tokens: 1024.

Anthropic requires budget_tokens to be at least 1024 and less than max_tokens. Anthropic Messages API reference.

Reject this combination, disable thinking with a warning, or raise the effective output limit. Add boundary tests for 1024 and 1025 tokens.

After those are addressed and you rebase on main, we should be good to 🚢 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

shellm sends invalid thinking and effort fields to the Anthropic API

2 participants