temp hot fix

[xzrderek]

Note

Adds resilience to LiteLLM chat forwarding.

• Implements 404 retry loop in handle_chat_completion with exponential backoff (up to 9 attempts; ~4.25 min max) using asyncio.sleep
• Introduces MAX_RETRIES_ON_404 and RETRY_BASE_DELAY_SECONDS constants; sets JSON headers explicitly
• Existing request/response handling and Redis insertion_id registration logic remain unchanged

^{Written by Cursor Bugbot for commit 9186f68. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[cursor]

Retry count mismatch: 9 retries instead of documented 8

Medium Severity

The comment states "8 retries" with delays of "1, 2, 4, 8, 16, 32, 64, 128 seconds" and total wait time of "~255 seconds", but MAX_RETRIES_ON_404 = 9 causes 9 retries. The 9th retry adds a 256-second delay, nearly doubling total wait time to ~511 seconds (~8.5 minutes). This could cause users to wait twice as long as intended when hitting persistent 404 errors.

Additional Locations (1)

• eval_protocol/proxy/proxy_core/litellm.py#L17-L20