Add streaming responses + litellm-sourced pricing

[drogers0]

Summary

Two features:

1. Streaming responses — the proxy can now stream Bedrock responses (SSE), via a transport migration to FastAPI + the Lambda Web Adapter on a Lambda Function URL (RESPONSE_STREAM), packaged as a container image. (HTTP API v2 cannot stream; Python Lambda needs LWA.)
2. litellm-sourced pricing — DEFAULT_PRICING is now generated from a vendored, Bedrock-filtered snapshot of litellm's community pricing data instead of the AWS pricing API (no runtime litellm dependency).

Pricing (litellm)

• scripts/gen_pricing.py reads scripts/vendor/litellm_model_prices.json (vendored, MIT-attributed, refresh procedure documented) and converts per-token USD → integer µUSD/1k.
• Cross-region variants derived bidirectionally for providers that actually have inference profiles (no phantom IDs).
• Coverage-regression gate: generation hard-fails if any model ID in scripts/vendor/supported_model_ids.txt would lose explicit pricing (prevents silent fallback to the over-count tier).
• Unknown models keep the budget-protective Opus-tier fallback — litellm's own behavior (silently $0 for unmapped models) is unsafe for a quota proxy, so it was deliberately not adopted.
• Embedding models (billable via /invoke) retained; zero-priced (rerank) dropped. compute_cost runtime path unchanged.

Streaming (transport + routes)

• New: lambda/proxy/app.py (FastAPI), deps.py, Dockerfile (python:3.12-slim + LWA), requirements.txt. handler.py removed. /converse, /invoke, /usage ported with identical auth/quota/billing semantics, error envelope, status codes, and structured-log events.
• New routes: POST /model/{id}/converse-stream and POST /model/{id}/invoke-with-response-stream — faithful SSE relay of Bedrock events.
  • Streams open eagerly so call-time errors return real 4xx/5xx (not 200 + error-event); mid-stream Bedrock EventStream error members (throttling, model-stream-error, model-timeout, …) become a terminal SSE error frame.
  • Usage captured from the stream tail (converse metadata; invoke message_start/message_delta, including cache tokens) and billed post-flight via the same compute_cost + write_usage path, disconnect-safe in finally.
  • Bytes-valued events are base64-encoded for JSON/SSE transport.
• IAM adds bedrock:InvokeModelWithResponseStream; CloudWatch alarms added (Throttles/Errors/ConcurrentExecutions).

Deployment / safe cutover (important — 2 live deployments)

The legacy API Gateway is retained behind enable_http_api (default true), so a single terraform apply is additive (APIGW stays live, Function URL added). Retire APIGW only after clients are cut over by setting enable_http_api=false. Container image requires an ECR push before apply — see the runbook in terraform/main/main.tf / terraform/main/README.md. Run per AWS profile (primary + ccc). Function URLs have no built-in throttling — set per-token limit_rps and rely on reserved concurrency.

Testing

• ruff check + ruff format --check clean; 131 tests pass, 3 e2e skipped (run with E2E=1 against a deployed stack); terraform fmt -check + validate pass.
• Adds a CI gate (.github/workflows/ci.yml) running ruff + pytest on push/PR.
• New: full FastAPI parity suite (test_app.py), streaming suite (test_streaming.py, 35 tests incl. eager-open errors, cache-token billing, EventStream error members, generator-level disconnect), pricing generator + fallback tests, and e2e streaming smokes.

Notes / follow-ups

• The streaming SSE relay is Bedrock-native (not OpenAI-shaped) — clients parse Bedrock event JSON.
• Deployment (terraform apply + ECR push + e2e against the live stack) is left to the operator per the runbook.