BAI-319 add session-savings statusline for model routing#109
Closed
imranq2 wants to merge 1048 commits into
Closed
BAI-319 add session-savings statusline for model routing#109imranq2 wants to merge 1048 commits into
imranq2 wants to merge 1048 commits into
Conversation
…pes and improve configuration handling
…n and dependency version
…idation and update snapshot cache methods for improved error handling
…ce CACHING.md documentation
…iguration for plugin support
…r count in Docker configuration
…marketplace' to 'plugins'
…egration and enhance .gitignore for cache management
…tings and increase Gunicorn worker count
…n options and enhance comments for better understanding
Records the 2026-07-14 investigation into the "0% until auto-compact" context-usage bug: symptom, root cause in both streaming translators, why the router's own logging didn't reveal which transport was actually live, the fix, and which other response paths were audited and confirmed already safe.
…ransport ran
MODEL_ROUTING_BEDROCK_TRANSPORT now defaults to "native" instead of
"mantle" — "mantle" becomes the manual opt-out instead of the default.
Mantle's own reliability (bare, undiagnosable internal_server_error
500s with no further diagnostic detail, as seen in a services-env
incident) makes native the better default now that both transports
have been proven out in production. Only the literal value "mantle"
(case-insensitive) opts back in; anything else, including a typo,
resolves to "native" rather than silently landing on the less
reliable transport. CodingModelRouter's own constructor default is
left at "mantle" since api.py always passes bedrock_transport
explicitly from this env var — flipping the constructor default too
would have silently changed which path ~20 existing tests exercise
for no behavioral benefit.
Every model-router-usage/model-router-errors record now also carries
a bedrock_transport field ("native"/"mantle") whenever
backend == "aws_bedrock". auth/api_type alone can't disambiguate this
today — native Converse dispatch is also reached via auth="aws",
api_type="openai" routes — which is exactly what made diagnosing the
services-env incident harder than it needed to be. record_error fills
this in automatically from CodingModelRouter._bedrock_transport (no
call-site changes needed); record_usage's callers pass it explicitly,
hardcoded per translator since each usage-tracking wrapper is
transport-specific by construction (_oai_stream_with_usage_tracking is
always Mantle, _converse_stream_with_usage_tracking and
BedrockNativeDispatcher are always native).
Removed the now-redundant MODEL_ROUTING_BEDROCK_TRANSPORT: native
override from docker-compose.yml (native is the default everywhere
now) and updated docs/coding_model_router.md and the original
native-transport design spec to reflect the new default and field.
…Bedrock dispatch paths - record_usage/session rollup now capture retry_count/total_retries across all six dispatch paths (Mantle + native Converse, streaming + non-streaming, plus the Anthropic-format Bedrock/passthrough path). - Native Bedrock Converse streaming now reports sse_event_count, matching Mantle's existing behavior - usage records for native-transport streams were silently missing this sanity field. - Dispatch pacing (_pace_bedrock_dispatch) was only wired into the unused Anthropic-format Bedrock route; now also applied to the Mantle and native Converse paths that actually serve the haiku/sonnet tiers in production, so concurrent requests no longer burst against Bedrock unthrottled.
BAI-299 Implement native AWS Bedrock Converse transport
…n and bedrock_transport field scope
BAI-315 Document Bedrock transport options
…hinking toggle on native transport
BAI-316 Add MODEL_ROUTING_QWEN_ENABLE_THINKING env var
…ey're recorded instead of surfacing as unhandled 500s
BAI-317 Make native Bedrock connect/read timeouts explicit env vars
…nippet Opening marker used <<< instead of >>>, matching docs/coding_model_router.md convention.
BAI-318 Move model_routing_guide.md from language-model-gateway-configuration
…gn docs router.py's and coding_model_router.md's docstrings claimed this router is deployed per-user/local, which justified an otherwise-spoofable custom-header user_id fallback. That assumption no longer holds (shared multi-tenant deployment) -- flagged as a known gap rather than silently left stale. Also fixes a pre-existing bug in pre-commit-hook (line 12 executed $GIT_PROJECT_ROOT as a command instead of cd-ing into it, breaking every commit in this repo since the initial commit) and adds the design spec and implementation plan for a session-savings statusline endpoint, to be implemented task-by-task on this branch.
…ons/{session_id}/savings
format_savings_line() now wraps all field-access and formatting in try-except, returning None silently on any TypeError, ValueError, AttributeError, or KeyError. This ensures the statusline command never prints a raw error string or traceback that would break Claude Code's UI, regardless of whether the gateway returns wrong field types (total_savings_usd as string, tier cost_usd as non-numeric, tiers as non-dict, etc.). Add 3 test cases covering the previously-crashing malformed-type inputs.
Adds a Prerequisites/Setup/Troubleshooting walkthrough (absolute path, chmod +x, manual test before wiring into settings.json, restart requirement) and ties MODEL_ROUTING_GATEWAY_URL to the same gateway_url local var as ANTHROPIC_BASE_URL in the claude-router() shell function, so the two env vars can't drift apart when swapping between local and production.
Corrects the collection-name default (usage_sessions -> model-router-sessions) in SessionSavingsReader/SessionSavingsRouter and their docstrings, plus stale usage_sessions and curl references in the session-savings-statusline design doc and plan doc. No runtime behavior change: api.py already passes the correct env-var-driven collection name to both reader and writer.
Owner
Author
|
Closing: created against the wrong repo by mistake (gh defaulted to this fork instead of icanbwell/language-model-gateway). Correct PR: see icanbwell/language-model-gateway. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GET /v1/model-routing/sessions/{session_id}/savings, a read-only endpoint returning a Claude Code session's cumulative cost savings (vs. Anthropic list price) from being routed through this gateway, broken down by model tier. Reads themodel-router-sessionsrollup collection thatusage_tracker.pyalready writes to (SessionSavingsReader,SessionSavingsRouter), mounted increate_app()alongside the existingCodingModelRouter.scripts/claude_code_statusline.py, a stdlib-only Claude CodestatusLinecommand that calls the new endpoint and renders💰 $X saved (haiku $... · sonnet $... · opus $...)in the terminal footer, plus adocs/model_routing_guide.mdwalkthrough for local setup.docs/superpowers/specs/2026-07-16-session-savings-statusline-design.md's "Security posture" section for the reasoning; response leaks only an aggregate dollar figure, never PHI/email/account identity.router.py/docs/coding_model_router.mdclaimed this gateway is deployed per-user/local (it's actually shared multi-tenant), which had been the stated justification for an otherwise-spoofable custom-headeruser_idfallback — corrected to flag this as a known gap needing a follow-up ticket; and a brokenpre-commit-hook(line 12 executed a variable as a command instead ofcd-ing into it) that blocked every commit in this repo since the initial commit.Test plan
pytest tests/gateway/routers/model_routing/test_session_savings_reader.py tests/gateway/routers/model_routing/test_session_savings_router.py tests/gateway/routers/test_session_savings_wiring.py tests/scripts/test_claude_code_statusline.py -v)docs/superpowers/plans/2026-07-16-session-savings-statusline.mdand.superpowers/sdd/progress.mdfor the review trailscripts/claude_code_statusline.pyinto a local~/.claude/settings.jsonper the new doc section and confirm the footer renders after a real Claude Code session against this gateway