Skip to content

Qwen3.5 XML parser loses schema strings and exposes undeclared tool calls #530

Description

@leehack

Confirmed behavior

Qwen3.5 XML tool-call parsing loses schema-declared string types and exposes undeclared function names through the public streaming parser.

With the selected inspect tool schema:

{"type":"object","properties":{"code":{"type":"string"},"options":{"type":"object","properties":{}},"items":{"type":"array","items":{"type":"string"}},"count":{"type":"integer"},"active":{"type":"boolean"},"empty":{"type":"null"}},"required":["code","options","items","count","active","empty"]}

The valid Qwen emission is:

<tool_call>
<function=inspect>
<parameter=code>
123
</parameter>
<parameter=options>
{}
</parameter>
<parameter=items>
[]
</parameter>
<parameter=count>
7
</parameter>
<parameter=active>
true
</parameter>
<parameter=empty>
null
</parameter>
</function>
</tool_call>

Expected code is string "123"; Dart returns integer 123. Replacing <function=inspect> with <function=unknown> still yields a public tool-call delta for unknown, even though only inspect is declared. Expected: no callable undeclared function, and the established safe final rollback/no partial tool leak behavior.

Provenance and causality

  • Reproduced using unmodified llamadart base 6c01a69af5bd13b338763f5f63cf16e482aa9fd6, before PR Fix structured tool-result rendering for Qwen templates #529, with JSON-string tool results that bypass Normalize structured typed tool results for Qwen chat templates #528's Map-rendering failure.
  • Also reproduced at PR Fix structured tool-result rendering for Qwen templates #529 head 2acefdc54fd29348ac89c84408a4ae8a639d22e2, for both typed Map and equivalent JSON-string result histories.
  • Exact affected template: Qwen3.5-0.8B-Q4_0 GGUF from ggml-org/Qwen3.5-0.8B-GGUF revision 8fea620810c4afa23dd6443f999a48574c1611a3; model SHA256 57d1997790d1744fba5b40a7317df71ea5e2acee28c47e78f0cce39c0703f8cf; template UTF-8 SHA256 273d8e0e683b885071fb17e08d71e5f2a5ddfb5309756181681de4f5a1822d80.
  • Production path: ChatTemplateEngine.renderChatCompletionStreamParser.parse, passing [inspect] as the actual tool definitions, with a previous assistant tool call and result in history. A thinking prefix is followed by character-split XML output; this is model-free parser validation, not a model-quality claim.
  • Unmodified pinned upstream llama.cpp commit b2e5e9b28b2484fbf94b543432ece638996a8b97, compiled common_chat_templates_apply + common_chat_peg_parse, same template/schema/history/emission: retains "code":"123"; undeclared function produces no tool calls.
  • Qwen's current Dart handler delegates to parseXmlToolCalls without schema-aware reconstruction/validation. The exact introducing commit has not been established. This is pre-existing, not caused by Fix structured tool-result rendering for Qwen templates #529 or suite Add portable validation suite and guarded Firebase/GCE runners #515.

Related closed #410 covered MiniMax M3/Muse/Laguna; this report is the verified Qwen3.5 path, not a claim that those repaired handlers regressed.

Acceptance

  • Preserve schema-declared numeric-looking strings, other scalar types, nested/empty containers through the public Qwen tool path.
  • Reject undeclared functions and malformed/invalid parameter structures without premature tool deltas; preserve correct final rollback.
  • Durable positive/negative tests tied to actual render/parser wiring and typed tool-result histories; test auto/required/none and thinking modes.
  • Compare matching emissions with pinned upstream and run VM/Chrome, template parity, compiled grammar, and exact affected-family model control as applicable.
  • Keep this separate from Fix structured tool-result rendering for Qwen templates #529's narrow template-input normalization. Its expanded strict regression oracles exposed this existing blocker; they must not be weakened to claim readiness.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions