Skip to content

feat(filter): add external metering filter for usage reporting and balance checks #577

Description

@noyitz

Description

Add an external_metering filter that integrates with an external metering/billing service for:

  1. Pre-request balance check: Queries the metering service to verify the tenant has available token budget before forwarding the request to the LLM provider. Rejects with 429 if budget is exhausted.
  2. Post-response usage reporting: After the response completes, reads token counts from filter_metadata (set by the existing token_count filter) and sends a CloudEvents-formatted usage report to the metering service.

Key design points

  • Single filter (not two like the Go/ext_proc predecessor) — leverages the existing token_count filter for usage extraction from all providers (OpenAI, Anthropic, Google, Bedrock, Azure)
  • Identity header capture: Captures configurable identity headers (default prefix: x-tenant-) containing username, group, subscription. Strips them before forwarding upstream.
  • Fail-open by default: If the metering service is unavailable, requests proceed normally (configurable to fail-closed)
  • Fire-and-forget usage reporting: Usage events are sent asynchronously via tokio::spawn — never blocks the response path
  • CloudEvents format: Standard CloudEvents 1.0 envelope with inference.tokens.used and inference.request.error event types
  • Uses CalloutClient from praxis-core for HTTP calls with circuit breaking and timeout

API contract

Balance check (pre-request):

GET {metering_url}/api/v1/customers/{username}/entitlements/{feature_key}/value?model={model}
Response: { "hasAccess": true, "balance": 9000.0, "usage": 1000.0, "overage": 0.0 }

Usage report (post-response):

POST {metering_url}/api/v1/events
Body: CloudEvent JSON with token usage data

Configuration

filter: external_metering
metering_url: "http://metering-service:8080"
timeout_seconds: 5
feature_key: "inference-tokens"
source: "ai-gateway"
fail_open: true
identity_header_prefix: "x-tenant-"

Pipeline ordering

filters:
  - filter: external_metering   # request: first, response: LAST
  - filter: token_count          # response: writes metadata before metering reads it
  - filter: router
  - filter: load_balancer

Motivation

Production AI gateway deployments need per-tenant token quota enforcement and usage tracking for billing/chargeback. Praxis has token extraction (token_count) but no way to report usage to an external metering or billing system.

This capability exists in a Go-based ext_proc predecessor and needs to be ported to native Praxis for the data plane migration.

Alternatives Considered

  1. Two separate filters (metering + headers-guard): Rejected because the identity headers are only consumed by metering, and combining them reduces pipeline complexity.
  2. External CloudEvents crate: Rejected — the CloudEvent envelope is simple enough to serialize inline with serde_json::json!().
  3. Synchronous usage reporting: Rejected — would add latency to every response. Fire-and-forget via tokio::spawn is appropriate since usage report failures should never affect the client response.

Metadata

Metadata

Assignees

Type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions