Skip to content

feat: Add Ollama environment variable support and fix compatibility - #12

Open
CodeNKasch wants to merge 7 commits into
masterfrom
feature/ollama-environment-parameters
Open

feat: Add Ollama environment variable support and fix compatibility#12
CodeNKasch wants to merge 7 commits into
masterfrom
feature/ollama-environment-parameters

Conversation

@CodeNKasch

@CodeNKasch CodeNKasch commented Oct 13, 2025

Copy link
Copy Markdown
Collaborator

Overview

This PR enhances Ollama integration with environment variable support and fixes compatibility issues, enabling seamless use of local LLMs without requiring Gemini API keys.

Key Features

🔧 Environment Variables

  • GIA_DEFAULT_MODEL: Set your preferred model (Ollama or Gemini)

    • Example: export GIA_DEFAULT_MODEL="ollama::llama3.2"
    • When set to an Ollama model, no Gemini API key is needed
    • Priority: -m flag > GIA_DEFAULT_MODEL > gemini-2.5-flash-lite
  • OLLAMA_BASE_URL: Connect to remote Ollama servers

    • Example: export OLLAMA_BASE_URL="http://192.168.1.100:11434"
    • Default: http://localhost:11434
    • Automatic URL normalization to /v1/ endpoint

🐛 Ollama Compatibility Fix

Fixed critical issue where Ollama's OpenAI compatibility layer couldn't handle multiple text parts in a single message:

  • Problem: Messages with multiple parts (role + file + prompt) would fail
  • Solution: Automatically merge multiple text-only parts into single text for Ollama
  • Benefit: Preserves Gemini's prompt caching while fixing Ollama compatibility

📚 Documentation

Updated README.md and CLAUDE.md with:

  • Quick setup examples for Ollama without API keys
  • Environment variable configuration
  • Model selection priority
  • Custom server configuration

🧪 Testing

  • Added comprehensive unit tests for URL normalization
  • Added tests for multi-part message merging
  • Added tests for message content preservation (Gemini caching)
  • Manual testing confirms all features working correctly

Usage Examples

# Set default model to Ollama (no Gemini API key needed)
export GIA_DEFAULT_MODEL="ollama::llama3.2"
gia "What is the capital of France?"

# Connect to remote Ollama server
export OLLAMA_BASE_URL="http://192.168.1.100:11434"
gia -m "ollama::llama3.2" "your prompt"

# Use with file inputs (multi-part merging works automatically)
gia -m "ollama::llama3.2" -f data.txt "Summarize this"

Technical Details

  • Renamed OLLAMA_MODEL to GIA_DEFAULT_MODEL for consistency
  • Removed unused created_at field from ConversationSummary
  • Removed unused notify_rust import
  • URL normalization ensures proper /v1/ endpoint format
  • Multi-part message merging preserves multimodal content (images, audio)

…guration

Introduce Ollama integration for local AI model execution. Enhance model selection logic to
prioritize Ollama when OLLAMA_MODEL is set. Update documentation to reflect new Ollama setup and
configuration options. Add environment variables for Ollama base URL and model. Improve error
handling and type clarity in Ollama client. Add comprehensive tests for model selection priority and
Ollama URL normalization.
@CodeNKasch
CodeNKasch requested a review from panjamo October 13, 2025 20:19
@CodeNKasch CodeNKasch self-assigned this Oct 13, 2025
@CodeNKasch CodeNKasch added bug Something isn't working feature and removed bug Something isn't working labels Oct 13, 2025
@CodeNKasch
CodeNKasch marked this pull request as ready for review October 13, 2025 20:21
This commit renames the OLLAMA_MODEL environment variable to GIA_DEFAULT_MODEL. This change aims to
unify the default model configuration across different AI providers, making the environment variable
naming more consistent and intuitive.

The GIA_DEFAULT_MODEL can now be used to specify either a Gemini model or an Ollama model by
prefixing the Ollama model name with ollama::.

Documentation and code examples have been updated to reflect this change. The priority of model
selection remains the same: -m flag > GIA_DEFAULT_MODEL > default Gemini model.
The created_at field is no longer needed in the ConversationSummary struct and has been removed.
This commit introduces several improvements to the Ollama integration:

 Enhanced Message Content Handling: The MessageContentWrapper now correctly identifies and handles
 ContentPartWrapper::Image and ContentPartWrapper::Audio variants. This ensures that when these
 media types are present, the message content is not unnecessarily merged into a single text part,
 which could lead to compatibility issues with Ollama.

 Streamlined Ollama Client Setup: The OllamaClient setup has been refined for better readability and
 maintainability. The custom base URL configuration now uses a more concise ServiceTargetResolver
 closure, making the client initialization cleaner.
This commit refactors how MessageContentWrapper and OllamaClient handle message parts to ensure
compatibility with both Gemini and Ollama.

Previously, multiple text parts in MessageContentWrapper were merged into a single text for Ollama,
which caused issues with Gemini's caching mechanism.

This change introduces the following:

 content_part_wrapper.rs: The to_genai_message_content function in MessageContentWrapper is
 simplified to always convert parts to their genai::chat::ContentPart equivalents. This ensures
 that Gemini receives the distinct parts necessary for its internal caching and processing. Tests
 are added to verify that multiple parts (text, file, clipboard, image) are preserved and correctly
 represented in the genai::chat::MessageContent.

 ollama.rs: A new private helper function merge_text_parts_if_needed is added to OllamaClient. This
 function checks if a ChatMessage contains multiple ContentPart::Text variants and, if so, merges
 them into a single MessageContent::Text. This addresses the Ollama compatibility issue where
 multiple text parts are not handled correctly by its OpenAI compatibility layer. This merging logic
 is applied only to messages being sent to Ollama. Tests are added to verify the merging behavior
 for single text, multiple text, and single text within a Parts structure.

By separating the concerns of data representation for Gemini and compatibility for Ollama, this
change improves the robustness and flexibility of the application's AI interaction layer.
@CodeNKasch CodeNKasch changed the title feat: ✨ Add Ollama environment parameter feat: Add Ollama environment variable support and fix compatibility Dec 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant