Add more sophisticated error handling for rate limits

[bdougie]

Overview

While Tier 3 provides higher rate limits (5,000 RPM, 80,000 TPM), we still need robust error handling to gracefully manage rate limit errors and provide a smooth user experience during high usage periods.

Current Rate Limits (Tier 3)

• Requests per minute: 5,000
• Tokens per minute: 80,000
• Requests per day: 10,000+

Requirements

1. Automatic Retry Logic

   • Exponential backoff for rate limit errors (429)
   • Maximum retry attempts configuration
   • Jitter to prevent thundering herd

2. Queue Management

   • Queue video processing requests when approaching limits
   • Priority queue for different request types
   • User feedback during queued states

3. Rate Limit Tracking

   • Track current usage against limits
   • Predictive throttling before hitting limits
   • Different strategies for different endpoints

4. User Experience

   • Clear messaging when processing is delayed
   • Progress indicators for queued requests
   • Graceful degradation options

Implementation Tasks

• Add rate limit error detection in openai.ts
• Implement exponential backoff with jitter
• Create request queue system for video processing
• Add rate limit headers parsing
• Implement circuit breaker pattern for repeated failures
• Add user-facing status messages
• Create fallback strategies (e.g., process fewer frames)
• Add monitoring/alerting for rate limit approaches

Code Examples

// Example retry logic
async function withRetry<T>(
  fn: () => Promise<T>,
  maxRetries = 3,
  baseDelay = 1000
): Promise<T> {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (error.status === 429 && i < maxRetries - 1) {
        const delay = baseDelay * Math.pow(2, i) + Math.random() * 1000;
        await new Promise(resolve => setTimeout(resolve, delay));
      } else {
        throw error;
      }
    }
  }
}

Success Criteria

• No failed requests due to rate limits during normal usage
• Graceful handling of traffic spikes
• Clear user communication during delays
• Monitoring visibility into rate limit usage

Related Files

• src/lib/openai.ts - OpenAI API calls
• src/lib/ai.ts - AI service router
• src/hooks/useVideoProcessor.ts - Video processing logic

Priority

High - Critical for production reliability and user experience