[Enhancement] Split LLM Routing for weak Ollama

[lundelltravis]

Hi! I’ve been experimenting with MuckScraper on a small local Ollama host (8GB VRAM) and ended up building a split‑routing system that lets the project run inference across both local Ollama and cloud LLMs (via OpenRouter). I wanted to share the changes in case this direction is useful for the main project — the architecture is modular enough that it could support any OpenAI‑compatible cloud provider with minimal adjustments.

Why:

Running everything through a local model wasn’t feasible on my hardware, but I still wanted fast, cheap inference for lightweight tasks. The solution was to route “simple” operations (classification, headlines, bias scoring) to Ollama, while sending heavier summarization and deep‑analysis tasks to a cloud LLM.

This keeps the pipeline responsive even on modest hardware while still enabling high‑quality summarization.

What changed:

I added a provider‑aware LLM client and updated the pipeline so each part of MuckScraper can choose the appropriate inference backend.

Key changes:

1. New split‑routing LLM client
   The file newsfetcher/llmclient.py was rewritten to support two providers:

• generate_text_ollama()
• generate_text_openrouter()
• get_embedding_ollama()
• get_embedding_openrouter()

Plus a unified generate_text(provider=...) interface and provider‑specific status/config checks.

Timeouts are configurable via .env (OLLAMATIMEOUT, OPENROUTERTIMEOUT).

2. Routing updates across the pipeline
   Lightweight tasks → Ollama
   Heavy summarization → OpenRouter

Specifically:

• Topic classification → Ollama
• Headline generation → Ollama
• Outlet bias scoring → Ollama
• Story summarization → OpenRouter
• Deep article analysis → OpenRouter
• Deep story reports → OpenRouter

This is all documented in IMPLEMENTATIONSUMMARY.md and LLMROUTING_CHANGES.md.

3. Environment configuration
   .env now supports:

OLLAMA_HOST=http://localhost:11434 OLLAMA_MODEL=gemma3:4b OLLAMA_TIMEOUT=120 OPENROUTER_TIMEOUT=120 OPENROUTER_HOST=https://openrouter.ai/api/v1 OPENROUTER_MODEL=openrouter/free OPENROUTER_API_KEY= LLM_PROVIDER=openrouter # Provider toggle: "ollama" (default, home machine), "gemini", "groq", or "openrouter" EMBEDDING_PROVIDER=openrouter EMBEDDING_MODEL=nomic_embed_text OPENROUTER_EMBEDDING_MODEL=text-embedding-ada-002

The OpenRouter section can be swapped for any OpenAI‑compatible cloud LLM (Anthropic, OpenAI, Groq, DeepSeek, etc.) by adjusting the base URL and model name.

4. Backward compatibility
   All existing function signatures remain intact.
   Code that doesn’t specify a provider still uses the default LLM_PROVIDER.

How it works:

The pipeline routes each LLM call based on the task type. This keeps local inference fast while offloading heavy work to the cloud.

If this direction aligns with your roadmap
I’d be happy to help adapt the routing layer. The architecture is intentionally simple so it can be extended to additional providers or models without major refactoring.

Thanks for building such a great project — this split‑routing approach has made it much more usable on constrained hardware.

[grregis]

Thanks for this, and sorry for the slow reply — this is a direction I want to take, and it's now on the list of things blocking a 1.0.

Before you spend more time adapting your version, a heads-up: llm_client.py changed quite a bit in 0.6.0, after you filed this. It picked up a fast/quality model tier, an optional fallback Ollama host with automatic recovery, and Groq as a third provider. A wholesale rewrite of the file would collide with all three, so here's the shape I think merges cleanly — mostly so you're not working against a design that's moved.

The tiers already exist; they just can't cross providers.

generate_text(prompt, tier=...) already splits calls into TIER_FAST (grouping confirmation, topic classification, headline generation, outlet bias — about 1,140 calls per run) and TIER_QUALITY (summaries and deep reports, about 65). Your routing table is essentially this split already. What's missing is that model_for_tier() picks a different model within one provider, and LLM_PROVIDER is a single global. So the plumbing for "which calls are cheap" is done — the gap is only that both tiers resolve to the same backend.

Config naming.

I'd add one optional var, LLM_FAST_PROVIDER, defaulting to LLM_PROVIDER when blank — same pattern as the existing OLLAMA_FAST_MODEL / GEMINI_FAST_MODEL / GROQ_FAST_MODEL, where blank means "same as the main one" and existing installs see no change. Your setup then becomes:

LLM_PROVIDER=openrouter
LLM_FAST_PROVIDER=ollama

That reads backwards from how you described it — the global ends up being the cloud provider, with Ollama as the exception — but it's the only ordering where someone who has only ever set LLM_PROVIDER keeps getting exactly what they get today.

OpenRouter can reuse the Groq path.

Rather than a separate generate_text_openrouter(), I'd generalize the existing Groq function: it already speaks OpenAI-compatible chat/completions, and OpenRouter is that same API at a different base URL. Pulling it into a shared _generate_text_openai_compatible(base_url, api_key, model) makes Groq and OpenRouter two configs over one implementation, keeps the 429/Retry-After retry that's already there, and gets DeepSeek/OpenAI/etc. for free — which I think is the "any OpenAI-compatible provider" outcome you were after.

One thing that won't work as written.

Your .env has EMBEDDING_PROVIDER=openrouter with OPENROUTER_EMBEDDING_MODEL=text-embedding-ada-002. Two problems: OpenRouter routes chat completions and doesn't serve embeddings, and ada-002 is 1536-dimensional while the embedding column is Vector(768), sized for nomic-embed-text. The dimension mismatch is the harder one — vectors from different models aren't comparable, so changing embedding provider isn't additive: it needs a migration and a re-embed of the entire corpus, or story grouping silently degrades. My plan is to keep embeddings on Ollama/Gemini and document the limitation, the way .env.sample already notes that Groq has no embedding models. Text generation is where the win is anyway — that's where the wall-clock goes.

How I'd split it up

1. Generic OpenAI-compatible provider (Groq refactor + OpenRouter). Self-contained, changes no routing behaviour.
2. LLM_FAST_PROVIDER + per-tier provider resolution. This is the part that closes this issue.
3. Making the health checks tier-aware — check_llm_status() and is_configured() currently return a single boolean that gates whole pipeline stages, which can't express "summaries are fine, classification is down."

If you'd like to pick something up, (1) is the cleanest place to start — it's independent of the routing semantics and useful on its own. (2) would be very welcome too if you'd rather go straight at the thing you built. Happy either way, and no obligation; I'm glad to do them myself if you'd rather just review.

Either way, thanks for the detailed writeup — the task-to-tier mapping in your issue matched where the pipeline's time actually goes, which made this much easier to scope.