Feature request: semantic classification filter backed by llm-d-sc

[cnuland]

Description

Summary

We would like to propose an optional filter that calls llm-d-sc for semantic classification, so Praxis can route and apply policy on what a request is, not only on its headers, path, and shape.

llm-d-sc is a small Rust service that classifies prompts and returns ranked, versioned
signals over gRPC. It is an incubating project under llm-d, and it was designed to sit
behind a gateway rather than be one.

A parallel proposal is open with vLLM Semantic Router (#2951). That one is a different shape, since they already classify in-process. Here the fit looks more natural: Praxis is the routing and enforcement layer, and llm-d-sc deliberately does neither.

The boundary

llm-d-sc is a signal producer, not a decision maker, and the wire contract enforces it:
the response type has no route or endpoint field, by design.

request -> Praxis filter -> llm-d-sc: classify(context)
                         <- ranked signals + model/tokenizer/taxonomy revisions
        -> Praxis applies policy, stickiness, guardrails, fallback, endpoint choice

Routing authority and session state stay entirely in Praxis. The classifier cannot
quietly start making routing decisions, because a route is unrepresentable on the wire.

What it gives Praxis

• Content-aware routing. A lookup goes to a small model, a proof to a large one. The
  shipped complexity taxonomy scores 0.97% on a held-out set.
• Content-aware policy. A sensitivity taxonomy (PUBLIC through NEVER_EGRESS) can
  inform whether a prompt may leave the boundary.
• Bring your own labels. Taxonomies are data: labels plus a few example texts each,
  loadable from a path with no rebuild. Routing to sales or support is a JSON file, not a
  fine-tune.
• Provenance. Every response carries the model, tokenizer, and taxonomy revisions plus
  a content digest of the loaded artifact.
• Lifecycle decoupling. A classifier revision change does not redeploy Praxis.

Cost in the request path

Stage	p50
network, ClusterIP hop	22 us
queue, bounded admission	5 us
tokenize	56 us
model forward	7.7 to 12.3 ms

Everything that is not the model is under 1% of latency, and a warm result cache answers in
about 0.09 ms without reaching the model. Sidecar versus separate Pod is latency-neutral.
Measured on one contributor's homelab.

Integration options

1. Example only. We publish a Praxis filter in our repository; nothing lands here.
2. Optional filter in filter/, feature-gated and off by default, attaching ranked
   signals to request context for later filters and routing rules.
3. A shared interface in protocol/, if Praxis wants a backend-agnostic notion of
   semantic signals with llm-d-sc as one implementation.

Our proto is small and we would rather adapt it to what a gateway needs. Long term goals will focus on low latency, and caching optimization. This is why we picked llm-d as a home, as I imagine this will turn into a mini-llm-d instance at scale where managing cache is critical and when dealing with systems that handle millions of requests per minute.

This should work for both session based intelligent routing and turn-by-turn. Ultimately hnadling turn-by-turn is the ultimate goal.

Questions

1. Is content-aware routing something Praxis wants a path for, or better left to filters
   maintained outside this repo?
2. Is the filter chain the right insertion point, and is there an existing pattern for a
   filter that makes a network call and enriches request context?
3. What failure semantics would you want? Past its admission bound llm-d-sc returns
   RESOURCE_EXHAUSTED rather than queueing without limit, so Praxis would need a defined
   fallback, most likely routing without the signal.

It is at version 0.1 and pre-1.0, so the wire contract may still change.

Motivation

Praxis can route and apply policy on what a request is, not only on
its headers, path, and shape.

Alternatives Considered

No response

Area

AI / Inference

Parent Epic (optional)

No response

[shaneutt]

maintainer +1, interested in a proposal to set the groundwork then we can build the filter as experimental and maintain it there while we also grow llm-d-sc 👍