ollama

[shenzo1]

Would you have plans to integrate ollama so that y-cli can run with local llms that work with MCPs such as llama3.2? It would offer the possibility for complete privacy that openrouter can't really provide.

[shcorya]

It's possible to use ollama by editing /home/user/.local/share/y-cli/bot_config.jsonl.

1. Add the bot using y-cli bot add, any name will do, this example uses "primary"

Your configuration at /home/user/.local/share/y-cli/bot_config.jsonl should then resemble this

{"name": "primary", "base_url": "https://ollama.internal/v1/", "api_key": "redacted", "model": "gemma3:12b"}
{"name": "default", "base_url": "https://openrouter.ai/api/v1", "api_key": "", "model": "anthropic/claude-3.7-sonnet", "openrouter_config": {"provider": {"sort": "throughput"}}, "mcp_servers": ["todo"]}

2. Using a text editor, delete the entry for the "default" bot
3. Replace the name of your new bot with "default" e.g. "primary" -> "default"

The bot_config.jsonl file should now look like this

{"name": "default", "base_url": "https://ollama.internal/v1/", "api_key": "redacted", "model": "gemma3:12b"}

4. y-cli init

[shenzo1]

Thanks for this! I was able to set up ollama inspired by what you've shared. Note that the api_key ollama was necessary for it to work for me.
{"name": "ollama", "base_url": "http://127.0.0.1:11434/v1", "api_key": "ollama", "model": "gemma3:12b", "prompts": ["mcp"], "mcp_servers": ["filesystem"]}

However, I am unable to use MCP tools with local models. I have tried gemma3:12b, llama3.2, llama3.2-latest, llama3-groq-tool-use, qwen2.5... Have you been successful in using MCPs?

[shcorya]

No, I haven't been able to get MCP working, also trying different models with no success.

[shcorya]

I also have tried to integrate ollama and MCP in Cherry Studio without success, so this issue does not seem to be exclusive to y-cli.

[shcorya]

I dug into this more and learned that the tool API that ollama uses is different from the OpenAI version. There's work being done to implement ollama compatibility, for example oterm, but this is not a y-cli issue and should probably be closed.

LocalAI might serve as an alternative to ollama in this case.

[shenzo1]

Yes that goes along with my own testing. I have had the most success (although not to the level of Claude Sonnet 3.7 or Quasar Alpha but not too far either) with qwen2.5-7b-instruct-cline-128k-q8_0. Only model I have tested that actually recognizes having MCP tools. It actually works pretty well considering it's size and speed.