Description
Add an external_metering filter that integrates with an external metering/billing service for:
- Pre-request balance check: Queries the metering service to verify the tenant has available token budget before forwarding the request to the LLM provider. Rejects with 429 if budget is exhausted.
- Post-response usage reporting: After the response completes, reads token counts from
filter_metadata (set by the existing token_count filter) and sends a CloudEvents-formatted usage report to the metering service.
Key design points
- Single filter (not two like the Go/ext_proc predecessor) — leverages the existing
token_count filter for usage extraction from all providers (OpenAI, Anthropic, Google, Bedrock, Azure)
- Identity header capture: Captures configurable identity headers (default prefix:
x-tenant-) containing username, group, subscription. Strips them before forwarding upstream.
- Fail-open by default: If the metering service is unavailable, requests proceed normally (configurable to fail-closed)
- Fire-and-forget usage reporting: Usage events are sent asynchronously via
tokio::spawn — never blocks the response path
- CloudEvents format: Standard CloudEvents 1.0 envelope with
inference.tokens.used and inference.request.error event types
- Uses
CalloutClient from praxis-core for HTTP calls with circuit breaking and timeout
API contract
Balance check (pre-request):
GET {metering_url}/api/v1/customers/{username}/entitlements/{feature_key}/value?model={model}
Response: { "hasAccess": true, "balance": 9000.0, "usage": 1000.0, "overage": 0.0 }
Usage report (post-response):
POST {metering_url}/api/v1/events
Body: CloudEvent JSON with token usage data
Configuration
filter: external_metering
metering_url: "http://metering-service:8080"
timeout_seconds: 5
feature_key: "inference-tokens"
source: "ai-gateway"
fail_open: true
identity_header_prefix: "x-tenant-"
Pipeline ordering
filters:
- filter: external_metering # request: first, response: LAST
- filter: token_count # response: writes metadata before metering reads it
- filter: router
- filter: load_balancer
Motivation
Production AI gateway deployments need per-tenant token quota enforcement and usage tracking for billing/chargeback. Praxis has token extraction (token_count) but no way to report usage to an external metering or billing system.
This capability exists in a Go-based ext_proc predecessor and needs to be ported to native Praxis for the data plane migration.
Alternatives Considered
- Two separate filters (metering + headers-guard): Rejected because the identity headers are only consumed by metering, and combining them reduces pipeline complexity.
- External CloudEvents crate: Rejected — the CloudEvent envelope is simple enough to serialize inline with
serde_json::json!().
- Synchronous usage reporting: Rejected — would add latency to every response. Fire-and-forget via
tokio::spawn is appropriate since usage report failures should never affect the client response.
Description
Add an
external_meteringfilter that integrates with an external metering/billing service for:filter_metadata(set by the existingtoken_countfilter) and sends a CloudEvents-formatted usage report to the metering service.Key design points
token_countfilter for usage extraction from all providers (OpenAI, Anthropic, Google, Bedrock, Azure)x-tenant-) containing username, group, subscription. Strips them before forwarding upstream.tokio::spawn— never blocks the response pathinference.tokens.usedandinference.request.errorevent typesCalloutClientfrom praxis-core for HTTP calls with circuit breaking and timeoutAPI contract
Balance check (pre-request):
Usage report (post-response):
Configuration
Pipeline ordering
Motivation
Production AI gateway deployments need per-tenant token quota enforcement and usage tracking for billing/chargeback. Praxis has token extraction (
token_count) but no way to report usage to an external metering or billing system.This capability exists in a Go-based ext_proc predecessor and needs to be ported to native Praxis for the data plane migration.
Alternatives Considered
serde_json::json!().tokio::spawnis appropriate since usage report failures should never affect the client response.