Vision Enabled Models

[NotThePassword]

First off, just want to say thank you for a great app! And the other thing I wanted to do was ask if you have thought about the models coming out with vision enabled and possibly added the ability to send images through the chat interface?

[Vali-98]

This has been requested many times, and it really just comes down to two issues:

1. ChatterUI lacks any media storage. We will need to figure out how to attach media to the app cleanly.
2. Vision models are very slow on mobile. Images are sent through an encoder which may not be supported by llama.rn and itself will take up about 200-300 tokens. Given the somewhat slow prefill speeds due to no NPU/GPU utilization, this makes running vision models non-viable, and honestly the results of said models are pretty poor due to their small sizes anyways.

I wont put it as a 'never add'. Models may get smarter and more efficient in the future, but its a 'not useful right now' situation.

[NotThePassword]

Understood. Thank you for taking the time to respond.

[danboid]

1. ChatterUI lacks any media storage. We will need to figure out how to attach media to the app cleanly.

I'm not sure which local model openassistant uses by default because it auto downloads it for you on first run but whatever it is it supports image analysis and worked quite well with the few images I tried with it on my Samsung Galaxy S10+ (running LineageOS).

Can chatterui not import the model used by openassistant? Maybe we'd need convert it first?

I'm not sure what you mean by "We will need to figure out how to attach media to the app cleanly?" ChatterUI already seems to allow loading images, its just that non of the models I've tried so far have successfully analyzed an image. "Take picture" doesn't work on my phone/OS. Again, can we not just copy whatever method openassistant used for this?

Very impressive app!

Thanks

[Vali-98]

I'm not sure what you mean by "We will need to figure out how to attach media to the app cleanly?" ChatterUI already seems to allow loading images, its just that non of the models I've tried so far have successfully analyzed an image. Again, can we not just copy whatever method openassistant used for this?

This post is a year old, I've implemented vision models since then. To use it for local mode, you will require the relevant mmproj model.

"Take picture" doesn't work on my phone/OS.

It requires permissions to do so.

[danboid]

Can you recommend a mmproj model that should run on my SGS10+ with 8 GB RAM?

Which model do you recommend for "general purpose" use and which one for coding (that runs under chatterui with 8GB RAM)? For coding I'm mainly concerned with C, python and shell scripting, if it matters.

I realise these ~4BN parameter models aren't going to compare with models that require 128GB VRAM etc but you could save me a lot of experimentation and testing. There are quite a few models available on huggingface and only so many hours in a day.

Thanks

[danboid]

I've not been able to find a version of gemma 4 that I've been able to successfully load into chatterUI yet

This one doesn't load for me ("Failed to load model" error)

https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf/tree/main

nor does this load:

https://huggingface.co/unsloth/gemma-4-E4B-it-qat-GGUF

nor this:

https://huggingface.co/unsloth/gemma-4-E2B-it-qat-mobile-GGUF/tree/main

I have been successful in loading this gemma 3 model:

https://huggingface.co/matrixportalx/gemma-3n-E2B-it-Q4_0-GGUF/tree/main

[tazztone]

gemma 3n will be possible to run?

[danboid]

I have successfully got gemma-4-E4B-it-Q4_0.gguf to load and run using chatterui 0.9.0 running under LineageOS 19 / Android 12 on my SGS 10+.

https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/tree/main

I would imagine this is probably the most advanced AI model I can currently run locally on my aging phone?

[danboid]

This post is a year old, I've implemented vision models since then. To use it for local mode, you will require the relevant mmproj model.

I have got gemma-4-E4B-it-Q4_0.gguf installed and running. I have also downloaded and installed mmproj-F16.gguf for the same model. chatterui seems to import mmproj-F16.gguf but if I load an image into a gemma 4 chat and and a prompt like "Describe this image" it says "I apologize but no image was provided"

I presume I've not loaded my mmproj file correctly or binded it to gemma 4 correctly. How do we get mmproj files to work under chatterui?