Skip to content

perf(provider): optimize anthropic prompt cache hit rate#18

Merged
bitifirefly merged 1 commit intodevelopfrom
backport/upstream-20260322-batch4
Mar 22, 2026
Merged

perf(provider): optimize anthropic prompt cache hit rate#18
bitifirefly merged 1 commit intodevelopfrom
backport/upstream-20260322-batch4

Conversation

@bitifirefly
Copy link
Copy Markdown

Summary

  • backport upstream LiteLLM prompt-cache optimization for Anthropic-compatible routes
  • improve _apply_cache_control to place two cache breakpoints:
    • system message
    • second-to-last message (conversation history prefix)
  • keep tool cache marker behavior (last tool definition)
  • update tests to verify both breakpoints and non-mutation of inputs

Validation

  • ruff check opencane/providers/litellm_provider.py tests/test_litellm_prompt_caching.py
  • pytest -q tests/test_litellm_prompt_caching.py tests/test_litellm_message_sanitize.py
  • pytest -q (full): 396 passed

@bitifirefly bitifirefly merged commit 388ad10 into develop Mar 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant