What
The CohereClient class at src/services/cohere_client.py:23 instantiates cohere.ClientV2 (the synchronous Cohere SDK client). All methods — generate(), generate_json(), and embed() — are declared async and awaited by callers, but the underlying .chat() and .embed() calls execute synchronously on the main asyncio event loop thread.
Why
Every LLM/embedding call freezes the entire FastAPI server for the duration of the Cohere API request (typically 1-10 seconds). Under concurrent requests, the server becomes effectively single-threaded — all other requests queue behind the blocking call. This makes the service unable to handle concurrent users and defeats the purpose of using FastAPI/async.
Scope
In scope:
- Replace
cohere.ClientV2 with cohere.AsyncClientV2 in the Cohere client
- Update all method calls to use
await on the async client methods
- Verify the async client has the same API surface
Out of scope:
- Changing the prompt structure or LLM parameters
- Adding caching or rate limiting
- Changing the FastAPI application structure
Acceptance Criteria
Technical Context
File to modify:
src/services/cohere_client.py — class CohereClient (line 23)
Current code:
self._client = cohere.ClientV2(api_key=self._api_key)
Fix:
self._client = cohere.AsyncClientV2(api_key=self._api_key)
The AsyncClientV2 has the same method signatures but returns awaitable coroutines. All callers already await the results, so no caller changes should be needed.
Callers:
src/services/generation.py — course generationsrc/services/quiz_generator.py — quiz generationsrc/services/feedback_engine.py — feedback generationsrc/services/embedding.py — embedding generationscripts/build_index.py — knowledge base index builder
What
The
CohereClientclass atsrc/services/cohere_client.py:23instantiatescohere.ClientV2(the synchronous Cohere SDK client). All methods —generate(),generate_json(), andembed()— are declaredasyncand awaited by callers, but the underlying.chat()and.embed()calls execute synchronously on the main asyncio event loop thread.Why
Every LLM/embedding call freezes the entire FastAPI server for the duration of the Cohere API request (typically 1-10 seconds). Under concurrent requests, the server becomes effectively single-threaded — all other requests queue behind the blocking call. This makes the service unable to handle concurrent users and defeats the purpose of using FastAPI/async.
Scope
In scope:
cohere.ClientV2withcohere.AsyncClientV2in the Cohere clientawaiton the async client methodsOut of scope:
Acceptance Criteria
CohereClientusescohere.AsyncClientV2instead ofcohere.ClientV2generate(),generate_json(), andembed()methods are truly asyncruff check src/ tests/passespytestpassesTechnical Context
File to modify:
src/services/cohere_client.py— classCohereClient(line 23)Current code:
Fix:
The
AsyncClientV2has the same method signatures but returns awaitable coroutines. All callers alreadyawaitthe results, so no caller changes should be needed.Callers:
src/services/generation.py— course generationsrc/services/quiz_generator.py— quiz generationsrc/services/feedback_engine.py— feedback generationsrc/services/embedding.py— embedding generationscripts/build_index.py— knowledge base index builder