Local hosted AI support

[joshuachris2001]

Would it be possible to have the ability to point the OpenAI Disciption feature to local self-hosted OpenAI compatible services such as Ollama? Or connecting to a llama.cpp server?

[chadclancy]

I believe adding support for a custom, locally hosted OpenAI-compatible server would significantly broaden flexibility and privacy.

Many users, especially developers and privacy-conscious individuals, prefer self-hosted LLMs (e.g., via Ollama, LM Studio, or local Docker instances). A dedicated endpoint selector would enable:

• Lower latency with local servers on the same network
• True data privacy, avoiding third-party API traffic
• Cost efficiency by bypassing per-token charges
• Model experimentation, allowing seamless switching between endpoints
• Offline resilience, by caching or queuing requests when connectivity is spotty

This feature would complement the apps current architecture without disrupting the privacy-first philosophy of on-device processing. Consider offering three modes: on-device, OpenAI API, and custom server endpoint.

Thanks for building something that already respects user privacy—this addition would make it even more adaptable for a wider range of use cases.