Error with vision + ollama (edit: available patch)

[alexandre-the-bee]

Hello,

I extensively tested OpenFox which appeared to be very stable while using ollama with many of my models; on a 12GBvram card. But consistently encountered an error related to vision.

The best way I found to reproduce is to build a random single html app with some graphics, then I use "Test UI" command, as soon as I can see the first screenshot taken by the model, something breaks. It appears for models with and without vision activated.

I see in the UI:

And in the console:
ERROR] LLM stream error {"error":"LLMError: HTTP 400: {\"error\":\"json: cannot unmarshal array into Go struct field ChatRequest.messages.content of type string\"}"}

I am not yet able to do any pull request yet, nevertheless I asked to investigate in openfox v2.0.128 with my ollama qwen3.5:9b setup. Food for thoughts below.
Thank you.

Diagnosis

Root cause

The error is thrown by your Go-based LLM backend (vLLM, SGLang, or OpenCode Go), not by OpenFox itself. Go's JSON decoder rejects the request because OpenFox is sending messages[].content as an array, but the Go server expects a string.

text

json: cannot unmarshal array into Go struct field ChatRequest.messages.content of type string

How it happens

OpenFox's convertMessages/convertAttachment (src/server/llm/client-pure.ts) sends content as an array of content blocks (text + image_url) when a message contains an image and the model is flagged as vision-capable:

typescript

if (modelSupportsVision) {
  return [{ type: 'image_url', image_url: { url: attachment.data } }]
}

This array is only acceptable to OpenAI/Anthropic-style backends. The Go backends (vllm, sglang, opencode-go) POST to an OpenAI-compatible /chat/completions endpoint that declares content as a string, so Go rejects the array.

Why it triggers on the first screenshot in Test UI

During Test UI the agent takes a screenshot (via read_file on the image path, or attaching it). When that screenshot is sent as an image attachment with your vision-capable model, OpenFox builds an array → the Go backend 400s.

Your config confirms this: you're using Ollama with several vision-capable models (qwen3.8, gemma4, omnicoder-2, etc.), and the Ollama/OpenCode Go path also lacks a native multimodal endpoint.

Proposed fix — pick one

Option A (recommended, minimal & safe): When the backend is a Go backend that requires string content, convert the image attachment to the existing text-placeholder fallback instead of an image_url block. This reuses the already-working vision-fallback path and costs nothing.

Option B (bigger): Add a native multimodal endpoint for the Go backends (/api/chat with images[]), mirroring the existing Ollama native client. More work, more risk.

Before I build, I need to confirm

1. Which backend/model are you actually running right now? (vLLM/SGLang/OpenCode Go + which vision model?) — this determines whether Option A fully fixes it.
2. Does the screenshot enter as an attachment or via read_file? (If it's purely read_file, the content is a string and shouldn't trigger this — I want to confirm the exact trigger path.)

[parmentf]

I got the same problem, using a ollama on glm-5.3-flash:cloud, so it's not only on local models.

openfox 2.0.132
ollama 0.33.1

[alexandre-the-bee]

I managed a fix (code change+test); I picked option B which is better for native multimodal support for Ollama.
Now works OK on my side with a basic model such as: qwen3.5:9b + vision fallback qwen3.5:0.8b
Also I enclosed the Vision Fallback settings (picture below) which is a bit more detailed vs the doc's screenshot docs/screenshots/vision-fallback.png currently in v2.0.132.

If someone can test the enclosed patch, and can confirm I could do a pull request because I lack confidence.

src/server/llm/ollama-native-vision-fix-openfox-v2.0.132.patch