feat: migrate Inference API to FastAPI router system #4755
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update demo script to use the newer LlamaStackClient and Agent API instead of the manual OpenAI client approach. Changes: - Switch from OpenAI client to LlamaStackClient - Use Agent API for simplified RAG implementation - Auto-select models with preference for Ollama (no API key needed) - Reduce code complexity from ~136 to ~102 lines - Remove manual RAG implementation in favor of agentic approach This provides a cleaner, more modern example for users getting started with Llama Stack.
Simplify the Ollama model selection logic in the detailed tutorial. Changes: - Replace complex custom_metadata filtering with simple ID check - Use direct 'ollama' in model ID check instead of metadata lookup - Makes code more concise and easier to understand This aligns with the simplified approach used in the updated demo_script.py.
Update the agent examples to use the latest API methods. Changes: - Simplify model selection (already applied in previous commit) - Use response.output_text instead of response.output_message.content - Use direct print(event) instead of event.print() for streaming This aligns the tutorial with the current Agent API implementation.
Modernize the RAG agent example to use the latest Vector Stores API. Changes: - Replace deprecated VectorDB API with Vector Stores API - Use file upload and vector_stores.create() instead of rag_tool.insert() - Download files via requests and upload to Llama Stack - Update to use file_search tool type with vector_store_ids - Simplify model selection with Ollama preference - Improve logging and user feedback - Update event logging to handle both old and new API - Add note about known server routing issues This provides a more accurate example using current Llama Stack APIs.
Fix conformance test failures by explicitly defining both application/json and text/event-stream media types in the 200 responses for streaming endpoints (/chat/completions and /completions). Changes: - Updated fastapi_routes.py to include explicit response schemas for both media types - Regenerated OpenAPI specs with proper 200 responses - Regenerated Stainless config This fixes the "response-success-status-removed" conformance errors while maintaining the dynamic streaming/non-streaming behavior. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
r-bit-rry
requested review from
ashwinb,
bbrowning,
cdoern,
ehhuang,
franciscojavierarceo,
leseb,
mattf and
raghotham
as code owners
meta-cla
bot
added
the
CLA Signed
|
This pull request has merge conflicts that must be resolved before it can be merged. @r-bit-rry please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork |
leseb
reviewed
Jan 28, 2026
Collaborator
|
@r-bit-rry a lot of failures in the tests. |
r-bit-rry
force-pushed
the
router-inference
branch
from
18f4dd0 to
c9a7d39
Compare
Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
leseb
approved these changes
Jan 29, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR migrates the Inference API to the FastAPI router system, building on the work done in PR #4191. This continues the effort to move away from the legacy
@webmethoddecorator system to explicit FastAPI routers.Changes
Implementation Details
llama_stack_api/inference//v1alpha/inference/rerankendpoint is properly configured in the Stainless configThis represents an incremental migration of the Inference API to the router system while maintaining full backward compatibility.
Test Plan
Co-authored-by: Gerald Trotman gtrotman@redhat.com (@JayDi11a)
This PR supersedes #4445 with a clean, rebased history.
BREAKING CHANGES: update OpenAIEmbeddingsRequestWithExtraBody to support token array