Skip to content

feat: add base64 vision encoding for Local LLM agents - #2045

Open
Maurice-aXeTech wants to merge 8 commits into
frappe:developfrom
Maurice-aXeTech:ai-ollama-fix-vision
Open

feat: add base64 vision encoding for Local LLM agents#2045
Maurice-aXeTech wants to merge 8 commits into
frappe:developfrom
Maurice-aXeTech:ai-ollama-fix-vision

Conversation

@Maurice-aXeTech

@Maurice-aXeTech Maurice-aXeTech commented Feb 23, 2026

Copy link
Copy Markdown

fixes #2207

When using Ollama or other local LLMs with a vision-capable model, images uploaded to a bot DM were previously only sent as a file path (e.g. [User uploaded an image: /files/receipt.jpg]), which local models cannot fetch. This PR encodes images as base64 data URLs so vision models can actually read them.

Changes:

  • ai.py: Added _build_vision_content() helper that base64-encodes images into OpenAI-compatible content blocks
  • ai.py: Both image-handling paths in process_message_with_agent() now use vision content for Local LLM
  • agents_integration.py: Fallback path now handles list content (vision) vs string content
  • raven_bot.json + TypeScript types: Added enable_vision_base64 field (Check, default off)
  • Bot settings → AI tab: Toggle to enable/disable vision encoding, only shown when Local LLM is selected
image

Before:

image

After:
image

@Maurice-aXeTech

Copy link
Copy Markdown
Author

@nikkothari22 What do I need to do for a review of this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add base64 vision encoding for Local LLM agents

1 participant