[Please vote] New feature: support other LLMs that provide OpenAI-compatible API

[eslavnov]

Hypothetically speaking, it should be very easy to add support for custom base_url and I guess it will work out of the box? This should allow us to connect something like deepseek or ollama.

I use local OpenAI so it was not included in the first release, but I am happy to implement and test it if there is enough demand, so please comment/vote.

[eslavnov]

Open questions:

1. Will streaming tokens work? (probably yes)
2. Will tool calls work? (depends on a model? how do we handle those that do not support it?)

[andywarburton]

I use Ollama on my local machine and if I use a model that doesn't support tools I normally just get a response "this model doesn't support tools". I don't see why this needs to behave any differently.

[eslavnov]

Run some quick and dirty tests yesterday. Replacing base_url works, I am getting responses from the Ollama. The streaming behaves funky, something needs adjusting because I get back only the first sentence. So 100% doable, I just need to find the time to debug it and fix it.

[RobMeades]

This would be great as I'd then be able to use TTMG with https://localai.io/.

[jeroenterheerdt]

I'd like to see this to be able to use OpenAI on Azure.