Conversation
…oviders Steps 1b–4a of the on-prem/Azure provider work: - 1b: Wire src/memory/vectordb.py (chatbot conversation-memory store) into the same provider pattern — new ConversationVectorStoreBase interface, create_conversation_vector_store() factory, CHAT_VECTOR_STORE_PROVIDER env var (kept separate from RAG's VECTOR_STORE_PROVIDER — two distinct indexes by design, per metadata shape). - 2: LLM_PROVIDER=azure_openai / EMBEDDING_PROVIDER=azure_openai. Uses the openai SDK's AzureOpenAI client (already installed, no new dependency) — OpenAIModel/OpenAIEmbedder are duck-type compatible with it unchanged. build_azure_openai_client() in model.py is shared with AgenticChatbot's tool-calling loop, which previously hardcoded OpenAI regardless of LLM_PROVIDER (a real gap, now fixed — it also now fails loud instead of silently ignoring LLM_PROVIDER=huggingface, since that provider can't support this agent's function-calling flow). - 3: MEMORY_PROVIDER=azure_redis. Azure Cache for Redis is Redis protocol-compatible, so RedisMemory needed zero changes — the factory just validates and passes through a rediss:// (TLS) AZURE_REDIS_CONNECTION_STRING. - 4a: VECTOR_STORE_PROVIDER=azure_search (RAG chunks index). New AzureSearchVectorStore(VectorStoreBase) in vector_store.py: creates the index on first use, reset() drops/recreates it matching Chroma's semantics, and query() translates Azure's response into the exact same nested-list shape Chroma's query() returns (retrieval.py's _retrieve() depends on that specific structure regardless of backend). The azure-search-documents SDK is imported lazily inside the class so Chroma-only deployments never need it installed, even though it's now in requirements.txt unconditionally. Verification (no live Azure resources available yet): - Existing pytest suite passes throughout. - Every new branch dry-tested: missing-credential paths raise clear RuntimeErrors naming the exact env var needed; provided-credential paths construct real client/store objects correctly (verified via actual SDK introspection, not assumption). - Default on-prem providers live-regression-tested against the running Docker stack after every change — chat, RAG upload, and RAG query all produce identical results to before this work. Remaining: step 4b (conversation-memory Azure AI Search index) and step 5 (integration/hardening, Compose profiles) — see CLAUDE.md's provider selection section for current status of each provider. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015kKU7Z2PgrEgifi5vaMnCN
## Summary
This commit adds three major enhancements to Cortex:
### 1. Comprehensive Test Suite (128+ tests)
- tests/api/test_ratelimiter.py - Rate limiter tests (30+ cases)
- tests/api/test_controller.py - HTTP controller tests (30+ cases)
- tests/business/core/test_model.py - LLM factory and implementations (25+ cases)
- tests/business/core/test_embedding.py - Embedding providers (25+ cases)
- tests/business/core/test_cost.py - Cost calculation (45+ cases)
- tests/business/core/test_live_data.py - Live data providers (35+ cases)
All tests passing with coverage of edge cases, error handling, and real-world scenarios.
### 2. Cost Tracking Metrics
- src/business/core/cost.py - Cost calculator for LLM and embedding API calls
- Pricing tables for OpenAI and Azure OpenAI models
- New Prometheus metrics:
- chat_cost_total{model}
- embedding_cost_total
- embedding_requests_total
- 6 new Grafana dashboard panels showing costs per model, total costs, and trends
- Automatic cost recording in API controller
### 3. Live Data Integration
- src/business/core/live_data.py - Live data provider interface
- Implementations: Mock, DuckDuckGo, NewsAPI providers
- web_search tool added to agentic chatbot
- LLM can now fetch real-time information (news, web search)
### Documentation & Branding
- Updated CLAUDE.md with comprehensive sections on testing, cost tracking, live data
- Reorganized .env.example with clear sections for all configuration
- Added IMPROVEMENTS_SUMMARY.md with implementation details
- Updated README.md with Cortex branding and repo rename instructions
- Updated monitoring labels to reference "Cortex" (app name) vs "personal-chatbot" (repo)
### Files Changed
- 6 new test files (700+ lines of tests)
- 2 new implementation modules (350+ lines)
- Updated metrics, controller, chatbot, dashboard, documentation
- All changes backwards compatible
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
API:
- New GET /sessions and DELETE /sessions/{session_id} endpoints, with
ListSessionsResponse / DeleteSessionResponse DTOs and a
ChatHistoryManager.delete_session() that clears messages before the
session row.
- send_message now re-raises HTTPException before the generic handler.
Without it the 400 for an empty message was caught by `except Exception`
and re-reported to the client as a 500.
Live data:
- DuckDuckGoSearchProvider switches from the public api.duckduckgo.com
endpoint to the duckduckgo-search package. The public API only returns
disambiguation topics, so real queries came back empty.
- NewsAPIProvider sorts by relevancy instead of publishedAt. This tool
answers questions, and publishedAt returns whatever matched most
recently rather than most closely.
DTOs:
- Migrate Pydantic `class Config` to `model_config = ConfigDict(...)`.
pytest.ini escalates DeprecationWarning from src.* to an error, so the
v1-style config was failing the suite.
Frontend:
- Session list with delete in the sidebar, wired to the new endpoints.
- CSS refresh.
Tooling:
- Add pytest.ini (asyncio_mode = strict, deprecations from src.* fail).
- Pin pytest, pytest-asyncio, duckduckgo-search.
- Record test_results.txt / test_details.txt for the 158-test run.
Suite: 158 passed on Python 3.11.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017S9zmh5R6ZShWiLHdYCC77
Unit tier (477 tests, no network, no API keys, ~20s): - tests/conftest.py: shared fakes for the embedder, conversation vector store, Redis memory and OpenAI client. - AgenticChatbot: tool dispatch, the tool-calling loop, the three-way persistence fan-out (Redis, Chroma, SQLite), provider guards. - Vector stores: Chroma and Azure AI Search, including the Azure-to-Chroma response-shape contract that _retrieve() depends on. The Azure SDK is faked into sys.modules, so azure-search-documents is not required. - RAG: retrieval, ingestion (chunk-id stability, table sections, batched index builds), query_rag / ingest_pdfs entry points and their metrics. - Prompt builders, including grounding rules and date injection. - Memory: RedisMemory, LongTermMemory, ChatHistoryManager, ResponseCache, ChromaVectorDB. - Session endpoints: validation, 400/404/500 mapping, and guards against 4xx errors being swallowed into 500s. Eval tier (35 tests, marked `eval`, deselected by default): - Retrieval quality: recall@1/@3, MRR, re-ranker lift, embedding sanity. - Hallucination: groundedness, refusal on unanswerable questions, and an LLM-as-judge with a calibration test for the judge. - Latency: per-stage p50/p95 with stage attribution. - Golden set describes a fictional company, so a correct answer without retrieval means the model is fabricating. - TestRerankerGate pins a known finding: the cross-encoder emits raw logits, so min_score=0.15 acts as sigmoid(0.15) ~ 0.54. Config and docs: - pytest.ini registers the `eval` marker and deselects it by default. - CLAUDE.md and tests/TESTING.md document both tiers. - .gitignore now covers every .env variant (keeps .env.example tracked). Suite: 477 passed, 35 deselected on Python 3.11. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C11BstLgv7gYH1qeJmhnmD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Steps 1b–4a of the on-prem/Azure provider work (continuing the foundation from the previous PR):
src/memory/vectordb.py(chatbot conversation-memory store) into the same provider pattern established for the RAG vector store: newConversationVectorStoreBaseinterface,create_conversation_vector_store()factory,CHAT_VECTOR_STORE_PROVIDERenv var (kept separate from RAG'sVECTOR_STORE_PROVIDER— two distinct indexes by design, since their metadata shapes differ).LLM_PROVIDER=azure_openai/EMBEDDING_PROVIDER=azure_openai, using theopenaiSDK'sAzureOpenAIclient (already installed, no new dependency). Also fixed a real gap:AgenticChatbot's tool-calling loop previously hardcoded OpenAI regardless ofLLM_PROVIDER— it's provider-aware now, and fails loud (instead of silently ignoring the setting) ifLLM_PROVIDER=huggingface, since local models can't do this agent's function-calling.MEMORY_PROVIDER=azure_redis. Since Azure Cache for Redis is protocol-compatible,RedisMemoryneeded zero changes — just arediss://(TLS) connection string.VECTOR_STORE_PROVIDER=azure_search. NewAzureSearchVectorStore(VectorStoreBase): auto-creates the index on first use,reset()matches Chroma's drop/recreate semantics, andquery()translates Azure's response into the same nested-list shape Chroma returns (a real constraint —retrieval.pydepends on that exact structure). The SDK is imported lazily so Chroma-only deployments never need it installed.Remaining: step 4b (conversation-memory Azure AI Search index) and step 5 (integration/hardening, Compose profiles).
Test plan
🤖 Generated with Claude Code
https://claude.ai/code/session_015kKU7Z2PgrEgifi5vaMnCN