Skip to content

Non-streaming responses leak <think> tags into content for qwen3.5, qwen3.6-moe, qwen3vl and qwen3.5-omni #739

Description

@Javinator9889

Summary

On the non-streaming OpenAI path, models whose parse_nstream_content does not
split the think block deliver the reasoning inside content instead of
reasoning_content, so the client shows raw <think> tags as the answer. When
generation is truncated mid-thought, the partial thought becomes the entire answer.

Affects qwen3.5, qwen3.6-moe, qwen3vl and qwen3.5-omni — all
reasoning-capable. Reproduced on main at 92f3f13.

Environment

  • OS: Ubuntu 26.04.1 LTS (kernel 7.0.0-31-generic)
  • NPU driver: NPU FW 1.1.2.64, amdxdna 0.7
  • flm version: FLM v1.0.6
  • flm validate:
[Linux]  Kernel: 7.0.0-31-generic
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.7
[Linux]  Memlock Limit: infinity

Reproduce

flm serve qwen3.5:9b
curl -s localhost:52625/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen3.5:9b",
  "messages": [{"role":"user","content":"What is 5 plus 6? Think briefly."}],
  "max_tokens": 400, "think": true
}' | jq '.choices[0].message'

Observed on 92f3f13 — no reasoning_content key, thought in content:

finish=stop    reasoning_content=False  <think>_in_content=True
content: '<think>\nThinking Process:\n\n1.  **Analyze the Request:**\n    *   '

Same with "max_tokens": 60 (finish=length), where the truncated thought is
the whole answer.

Cause

build_nstream_response takes reasoning_content from parse_nstream_content,
and only some models populate it:

splits reasoning does not
gemma4_12b, gemma4e, gpt_oss, lfm2, llama3, nanbeige, qwen3 qwen3_5vl, qwen3_6_moe, qwen3vl, qwen3_5_omni

The streaming path is unaffected: its parser switches to REASONING on
<think> and stays there, so streamed responses split correctly. Only the
non-streaming path is wrong.

Impact

Agent clients render the thought as the assistant's reply. A truncated thought
reads as the prompt being echoed back ("Thinking Process: 1. Analyze the
Request:
The user wants me to ..."), which looks like the model looping.

Fix

PR follows: split the block once in build_nstream_response so every model
reports reasoning the same way, and route an unclosed (truncated) block to
reasoning_content with empty content. No-op for models that already
populate it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions