Skip to content

Enterprise routing: select interchangeable deployments using bounded runtime signals #148

Description

@tcballard

Why

Weighted pools provide deterministic distribution, but equivalent deployments can differ materially in current latency, throughput, capacity, cost, and failure rate.

Scope

  • Collect bounded rolling latency, time-to-first-token, throughput, availability, rate-limit, and capacity signals.
  • Add deterministic, policy-bounded selection strategies for interchangeable deployments.
  • Support price ceilings and stable tie-breaking.
  • Keep shared fleet health authoritative where configured.
  • Expose selection reasons in route receipts and prompt-free telemetry.
  • Fall back to stable weighted selection when observations are absent or stale.

Acceptance criteria

  • Runtime signals never choose a different model tier, privacy boundary, capability class, or named route.
  • Pinned destinations retain pin semantics.
  • Stale, sparse, or conflicting telemetry produces deterministic behaviour.
  • Selection remains bounded under metric floods and high-cardinality deployment catalogs.
  • Tests cover latency, throughput, cost, rate-limit, capacity, stale-data, and tie boundaries.

Parent: #146

Invariant

The scored model decision remains offline, deterministic, keyless, and explainable.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions