feat(cache): Add proxy-controlled function to keep cache warm
Most LLM providers use heavily discounted rates for cached reads/writes. But those typically come with TTL/time expiration limits.
Problem:
- in typical workflows agents run test suite multiple times (or at least once) per session. Some of such runs may take longer than the cache TTL. In such cases waiting for long running commands to finish causes full cache miss on subsequent tool output turn
Solution:
- we already have service which can classify tool call types, it should be used to detect tool calls to tools like bash/exec and similar and monitor how long they take
- we should construct a catalog of typical cache TTL params per inference provider or model (this is one off task, must not be run in hot code paths)
- if we discover that session is stuck and cache is close to invalidation due to the TTL, proxy should execute some dummy opetation in the background, in context of given session, which would incur minimal charge on input and output tokens, just to keep the cache warm to prevent full session miss which may be SLOW and COSTLY.
- should be configurable (default on, possibility to disable at server start or at level of specific session), should have configurable number of max "keep warm" actions before we give up
Notes:
- as some agents have such functionalities built-in, we should not try to re-invent the wheel but do a research phase first and search online for similar "keep cache warm" solutions and check for established patterns ad practices. Then we should take the best observed ones and consider use in our proxy, after adapting to the architecture
feat(cache): Add proxy-controlled function to keep cache warm
Most LLM providers use heavily discounted rates for cached reads/writes. But those typically come with TTL/time expiration limits.
Problem:
Solution:
Notes: