Image reading tool appears to be broken

[KiryunKang]

Summary

I found that the image reading tool does not seem to work correctly in OpenClaude. It fails when trying to read image files, even though the tool is expected to support them. Could you please check whether this is a known issue?

[kvnpmz]

Partially fixed in OpenClaude v0.1.8. Pasted images are working as expected, but reading image files (e.g., PNGs) still fails. Running npm install -g @gitlawb/openclaude@latest updates to this version.

[Vasanthdev2004]

Tagging as bug. @KiryunKang — are you still seeing this on the latest version? If so, could you share the model and error output?

[gnanam1990]

Hi @KiryunKang @kvnpmz — investigated this. The local file-reading path (FileReadTool.readImageWithTokenBudget in src/tools/FileReadTool/FileReadTool.ts:1100) works correctly: it reads the file, detects the format, resizes/compresses to fit the token budget, and emits an Anthropic-shape image content block.

The likely root cause is on the transport side, not the file-read side:

• Pasted images go into the conversation as content blocks on a user role message → the OpenAI shim translates them to image_url parts on a user-role message → all multimodal-capable providers accept this. This works.
• Read-from-disk images come back as content blocks inside a tool_result → the shim emits them as image_url parts on a role: tool message (verified by openaiShim.test.ts:4158). OpenAI's API now accepts this; many third-party OpenAI-compatible providers do not — Groq, OpenRouter routes to non-multimodal models, DeepSeek, Mistral, etc. silently drop or reject the image.

To confirm and scope the fix, could one of you share:

1. Provider (OpenAI, Groq, OpenRouter, DeepSeek, Mistral, NVIDIA NIM, Ollama, etc.)
2. Model (e.g., gpt-4o, llama-3.3-70b-vision, minimax-m2.5)
3. Exact error or symptom — does the model say "I can't see the image", does the API return a 4xx, or does the read just produce no useful output?

Knowing the failing provider lets me scope the fix correctly. The likely fix is: when a tool_result contains an image, split into a tool-role message with text only + a synthetic user-role message carrying the image — that works on all multimodal-capable providers but is currently a behavior change that needs provider-aware gating to avoid degrading OpenAI users.

If we don't hear back in ~7 days I'll close this as needs-info; happy to reopen / pick up the fix the moment we have a repro target.

[dev-marcel-anselmo]

@gnanam1990 Hello, I'm using Xiaomi Mimo 2.5 Pro and it's giving an error when pasting an image or requesting to analyze an image from the project. It returns this error: API Error: 400 {"error":{"code":"400","message":"Param Incorrect","param":"text is not set","type":""}}

[Vasanthdev2004]

Thanks for the concrete provider error. This looks like a provider-specific multimodal transport compatibility issue, not simply the local image-read path failing.

The Param Incorrect ... \text` is not set` response suggests Xiaomi MiMo's OpenAI-compatible endpoint may require a text part alongside image parts, or may not accept images in the same message shape that OpenAI accepts. That matters because pasted images and images returned from the file-read tool travel through slightly different message paths.

Could you share a few exact details so we can scope the fix safely?

1. OpenClaude version (openclaude --version)
2. Exact provider/base URL setup you are using for Xiaomi MiMo
3. Exact model name
4. Whether both cases fail: pasted image and image read from project file
5. The smallest repro command/prompt, plus whether there was any text in the same prompt as the image

Likely fix direction: add provider-aware handling for image content so providers that require user-role image parts or non-empty text fields get a compatible message shape, without regressing OpenAI-compatible providers that already accept the current shape.

[dolphprefect]

I have the same problem with DeepSeek V4.

I can provide some context here, since this happens all the time - I have to advice the agent to not ingest images or it breaks the session.

1. 0.12.0
2. https://api.deepseek.com/v1
3. deepseek-v4-flash / deepseek-v4-pro / deepseek-reasoner / deepseek-chat
4. Both cases fail.
5. "read this image: appicon.png"

Result:

└  API Error: 400 {"error":{"message":"Failed to deserialize the JSON body into the target type: messages[3]: unknown variant
     `image_url`, expected `text` at line 1 column
     182617","type":"invalid_request_error","param":null,"code":"invalid_request_error"}}

Only /rewind helps at this point.

[MikeTomlin19]

Opened PR #1410 with a scoped fix for the OpenAI-compatible image-only payload case where providers reject requests with text is not set.

This specifically targets the Xiaomi MiMo/OpenGateway-style path by ensuring image-only multipart content includes a small non-empty text part. It does not try to cover separate text-only provider image handling, which is being tracked elsewhere.