Design Trade-offs: Determinism vs LLM Routing

[jharjadi]

Design Trade-offs: Determinism vs LLM Routing

One of the core architectural decisions in this project is how to balance deterministic routing with LLM-assisted classification.

The Trade-off

Pure Deterministic Routing

Pros:

• Predictable behavior
• Zero LLM cost for routing
• Fast response times
• Easy to debug and test

Cons:

• Brittle keyword matching
• Poor handling of natural language variations
• Requires constant rule updates

Pure LLM Routing

Pros:

• Handles natural language well
• Adapts to variations automatically
• No manual rule maintenance

Cons:

• Adds latency to every request
• Increases cost significantly
• Non-deterministic behavior
• Harder to debug failures

Our Hybrid Approach

This project uses a two-tier system:

1. Fast deterministic check first - Keyword matching for obvious cases
2. LLM classification as fallback - For ambiguous or complex queries

See core/intent_classifier.py for implementation.

Why This Works

• Cost optimization: ~70% of queries hit deterministic rules (zero LLM cost)
• Quality: Complex queries still get intelligent routing
• Observability: Clear logs show which path was taken
• Tunable: Can adjust the balance based on production metrics

Open Questions

1. Should we add a confidence threshold?
   Currently, if deterministic matching fails, we always call the LLM. Should we have a "confidence score" that triggers human review for ambiguous cases?

2. Should routing be cached?
   Similar queries could reuse previous routing decisions. Trade-off: memory vs cost.

3. Should we support multi-intent queries?
   Currently, we route to a single agent. Should we support queries that need multiple agents?

Your Thoughts?

What trade-offs have you encountered in agent routing? How would you approach this differently?

---

Related Documentation:

• Intent Classification
• Agent Router
• Multi-LLM Strategy

[devonakelley]

Interesting trade-offs here. We went through a similar design process building Kalibr and landed on a different approach to the routing problem.

Instead of classifying intent first and then routing to a model, Kalibr routes the full execution path (model + tool + parameters) based on whether the agent's goal was actually achieved. Uses Thompson Sampling with Wilson scoring, so it balances exploration vs exploitation automatically. ~90% of traffic goes to the best known path, ~10% explores alternatives. When something degrades, rerouting happens in under 200ms.

The key insight for us was that the routing decision shouldn't just be "which model handles this query type" but "which model+tool+config combination actually succeeds at this specific task." Two gpt-4o calls with different tool configs can have wildly different success rates for the same goal.

Re: your open questions -- caching routing decisions is tricky because model performance drifts over time. We handle this with continuous exploration rather than caching. On multi-intent, we scope each Router to a single goal so multi-intent queries get decomposed upstream.

Happy to compare notes on the confidence threshold question if you're interested. We found that hard thresholds don't work well because what counts as "confident enough" varies per goal.