[Enhancement] AI service responds slowly with no streaming or caching

[zeemscript]

What's Happening

The AI service is responding slowly when users interact with AI powered features on the platform. Users experience noticeable delays when waiting for AI generated responses, which affects the overall user experience.

Current Behavior

When a user sends a request to the AI service, there is a significant delay before the response comes back. This makes the interaction feel sluggish and can lead users to think the feature is broken or unresponsive.

Technical Context

The service is built with FastAPI and uses Google's Gemini API for generating responses. The slow response time could be caused by several factors:

1. No response caching for similar or repeated queries
2. Synchronous processing that blocks while waiting for the AI model
3. No streaming of responses (users wait for the complete response)
4. Cold start issues on the hosting platform
5. Inefficient prompt construction or token usage

Possible Improvements

Some approaches to consider:

1. Implement caching for frequently asked questions or similar queries
2. Add response streaming so users see text appearing progressively
3. Optimize prompts to reduce token usage and response time
4. Add request queuing for better handling of concurrent users
5. Consider connection pooling or keep alive connections to the AI provider

Acceptance Criteria

The AI service should respond noticeably faster than it currently does. Ideally, users should start seeing response content within 1 to 2 seconds of sending their query. If full responses take longer, streaming should be implemented so users see progressive output rather than waiting for everything at once.

Measuring Success

Before and after response times should be measured and documented. The goal is to reduce perceived latency by at least 50% or implement streaming to provide immediate feedback to users.

[payfoxX]

I'd like to work on this issue.

I have experience working with FastAPI services and understand the importance of reducing both actual and perceived latency in AI applications. My approach would be to first benchmark the current request lifecycle to identify the biggest bottlenecks, then implement improvements that provide measurable gains rather than premature optimizations.

I plan to introduce response streaming so users receive output almost immediately, add caching for repeated or semantically similar requests where appropriate, and profile prompt construction and model calls to reduce unnecessary latency and token usage. I'll also ensure the implementation includes before-and-after performance measurements and remains compatible with the existing API while keeping the codebase maintainable.

I'll validate the improvements with tests and benchmarking to demonstrate the reduction in response time and improve the overall user experience.