feat: switchyard_route — Mixture-of-Models routing POC around NeMo Switchyard - #4
Draft
usize wants to merge 9 commits into
Draft
feat: switchyard_route — Mixture-of-Models routing POC around NeMo Switchyard#4usize wants to merge 9 commits into
usize wants to merge 9 commits into
Conversation
Port the switchyard-poc scaffold (4447055) onto the conventioned main, restructured to the template's crates/ layout (praxis-proxy#1, Track A of praxis-proxy/ai#758): - crates/switchyard-filters: [package.metadata.praxis-filters] auto-discovery marker, export_filters! registration, and a no-op experimental_placeholder filter with unit tests proving registration (replaced by switchyard_route in a follow-up commit, experimental#2). - crates/switchyard-server: discovery build.rs via praxis-ai-build-support, composing the git-pinned praxis-ai-proxy server (build_full_registry + run_server_with_registry) with discovered filters. - Toolchain/MSRV: bump 1.96 -> 1.96.1 (rust-toolchain, workspace rust-version, clippy.toml msrv) because the Switchyard 0.2.0 crates pin MSRV 1.96.1. Needs a maintainer decision on the template side. - deny.toml: re-add [sources] allow-git for praxis-proxy/ai (publish = false crates, git-pinned by switchyard-server). - docs/identity-metadata.md: the proposed identity metadata contract for the upcoming api_key_auth producer and the metering/budget consumers. Validated: build, test, clippy -D warnings, nightly fmt, rustdoc -D warnings, cargo machete. Assisted by Fable 5 Signed-off-by: usize <mofoster@redhat.com>
Implement praxis-proxy#2 (Track A item 2 of praxis-proxy/ai#758, discussion praxis#976): a decision-only Mixture-of-Models router embedding NVIDIA NeMo Switchyard's LlmTaskClassifier (Capability mode, =0.2.0, git tag v0.2.0 / commit 1fc9ab88) in the Praxis request path. Design (see crates/switchyard-filters/src/switchyard.rs module docs): - Two-phase pipeline: with BodyMode::StreamBuffer the buffered body hook runs first — parse body, detect wire format (OpenAI chat + Anthropic messages), decode to the Switchyard IR for the judge, drive run_stream to the routed Step::Decision, rewrite body model in place; the chosen cluster is stashed in filter metadata and applied to ctx.cluster in on_request. - Decision-only: the filter serves ONLY the judge CallLlm (via the server-shared SubRequestClient against an explicit judge endpoint) and drops the stream at the routed decision, before any answer call. - Host-owned no-downgrade guarantee: (1) every failure path passes the request through UNMODIFIED (never forces a tier); (2) an in-process session floor (DashMap + TTL, evicted on x-switchyard-session-final) clamps every decision to max(floor, decision) and optionally excludes the below-floor tier inside Switchyard. Switchyard itself cannot provide this: its session affinity is a first-decision-wins latch and its state is neither durable nor seedable. A durable floor store is a follow-up. - The tag->(cluster, model) table lives only in filter config; Switchyard sees abstract weak/strong/judge tags. Notable deltas discovered against the plan: - The pipeline's FilterEntry wrapper reserves the failure_mode key (parse_filter_config strips it), so the filter's knob is on_failure. - libsy's run_stream driver always offloads every model call to the step stream, so no RoutedLlmClient is attached to any target; the step loop serves the judge directly. - The algorithm is built once at from_config time (its session-state sweeper task must not be respawned per request). Also: allow MIT-0 (borrow-or-share via jsonschema <- switchyard-libsy) and ignore two unmaintained advisories pinned by the pingora-fork tree (RUSTSEC-2024-0388 derivative, RUSTSEC-2025-0134 rustls-pemfile) in deny.toml; add OpenAI/NeMo to clippy doc-valid-idents. Tests (45, hermetic in make test): config parse/validation, wire-format detection + IR decode + model rewrite for both formats, session-floor ratchet/TTL/eviction, judge endpoint parsing, and filter-level tests driving the real hooks against hand-built HttpFilterContexts with a scripted loopback judge stub over a real SubRequestClient — including: easy->weak, hard->strong, a strong session never downgrades when the judge later says weak, a judge failure leaves the body untouched, and session-final eviction. Validated: build, test, clippy -D warnings (workspace wall), nightly fmt, rustdoc -D warnings, cargo machete, cargo deny check, cargo audit, coverage 94.3% lines / 93.6% regions (gates: 90/80), single praxis-proxy-filter 0.5.2 in the tree. Assisted by Fable 5 Signed-off-by: usize <mofoster@redhat.com>
- docs/switchyard-route.md: design summary, full config reference (incl. the on_failure-vs-failure_mode structural-key note), the no-downgrade guarantee and its honest loss profile, the open-mode failure topology (pass-through cannot conjure a route: an LB-terminated chain returns 500 on judge outage; closed gives a deliberate 503), and a verified local demo transcript. - hack/switchyard-demo/: praxis.yaml (gateway :18080, judge :18091, weak :18092 / strong :18093 clusters) and stubs.py (stdlib-only judge + echo upstreams). Verified live: easy->weak with model rewrite, hard->strong, the session floor holding strong on a later easy turn, session isolation, and fail-open pass-through with the judge down. - switchyard-filters: declare test tokio runtime features as dev-dependencies. Assisted by Fable 5 Signed-off-by: usize <mofoster@redhat.com>
Replace the scripted judge with a real OpenAI-compatible judge endpoint: praxis.yaml.template is rendered by run-demo.sh (JUDGE_ENDPOINT / JUDGE_MODEL), upstreams.py stubs only the two tier clusters, and the guide carries a transcript recorded against a local Ollama llama3.2:3b judge — including the no-downgrade floor overriding a real weak verdict, and a thinking model (qwen3:8b) tripping the judge deadline into fail-open. Assisted by Fable 5 Signed-off-by: usize <mofoster@redhat.com>
…ntial
A hosted OpenAI-compatible judge (OpenAI, Together, Fireworks, …) needs a
bearer token, which the callout previously could not send. Add an optional
`judge.auth` block:
judge:
auth:
value_env: OPENAI_API_KEY # env var holding the secret (required)
header: authorization # default
scheme: Bearer # default; "" sends the raw value
The secret never appears in YAML — config names the environment variable
and the value is resolved once at filter construction, failing fast if it
is unset or empty. The resolved header value is marked sensitive so it is
redacted from tracing, and injected into the judge sub-request. Omit the
block entirely for a keyless judge (local vLLM/Ollama).
The demo runner gains a hosted-judge path: set JUDGE_KEY_ENV to the
variable name and run-demo.sh renders the auth block (dropping it when
unset) and exports the credential to the server; it also raises
RUST_LOG so successful routing decisions are visible in server.log.
Assisted by Opus 4.8
Signed-off-by: usize <mofoster@redhat.com>
The Capability classifier judges every turn — that is Switchyard's default and, until now, this filter's. But once a session is floored at the top tier, the no-downgrade clamp forces `strong` regardless of the verdict, so re-judging spends an LLM call to re-derive a foregone answer. Add an opt-in `session_floor.escalation_ratchet` (default false, matching Switchyard vanilla). When enabled, `route()` short-circuits above the judge callout on any turn whose floor is already `strong`: it routes `strong` directly and never dials the judge. This is the filter-side equivalent of Switchyard's own directional escalation latch (`AffinityRouter::with_latch_only(["strong"])`), expressed against the floor store the run_stream decision-only design already owns rather than adopting `AffinityRouter` (which keys on request metadata inside the classifier cascade we bypass). Tests use the scripted judge stub's connection count as ground truth: with the ratchet on, two strong turns capture exactly one judge request; off, both turns judge. Also correct the algorithm.rs comment that claimed Switchyard affinity cannot ratchet — it can, via with_latch_only. The demo enables the flag so turn 3 visibly holds strong with no judge call (grep 'ratchet held' server.log — three judge calls, not four). Assisted by Opus 4.8 Signed-off-by: usize <mofoster@redhat.com>
|
Non-conforming commit subjects (expected
Amend with |
|
PR too large: 3012 lines added (limit: 750, excludes Cargo files, tests, docs, examples, and benchmarks). Please split into smaller PRs. Add |
The Containerfile's dependency-cache stage copies each workspace
member's manifest and stubs its source before the warmup build. It
only listed experimental-probe, so once the workspace gained the
switchyard-filters and switchyard-server members, cargo could not
load the workspace manifest inside the builder ("failed to read
crates/switchyard-filters/Cargo.toml") and the Container job failed.
Copy and stub both new crate manifests (lib.rs for the filters lib,
main.rs for the server bin). The image still ships only the probe
binary; the switchyard sources stay stubbed in the builder.
Also fix a typo flagged by meta-lint: unparseable -> unparsable.
Verified: `docker build -f Containerfile .` succeeds.
Assisted by Opus 4.8
Signed-off-by: usize <mofoster@redhat.com>
Adding OpenAI/NeMo pushed the doc-valid-idents array past taplo's column width, so it must be expanded to one entry per line. Content is unchanged; this only satisfies `taplo fmt --check` in meta-lint. Assisted by Opus 4.8 Signed-off-by: usize <mofoster@redhat.com>
markdownlint-cli2 (meta-lint) enforces MD060 table-column-style "aligned": every row's pipes must line up with the header separator. The Required and Meaning columns had cells wider than the header, so the closing pipes drifted. Widen the header/separator to the widest cell and pad every row to match. Content unchanged. Verified: `markdownlint-cli2@0.23.2` over all tracked *.md is clean. Assisted by Opus 4.8 Signed-off-by: usize <mofoster@redhat.com>
praxis-bot
reviewed
Aug 20, 2026
praxis-bot
left a comment
praxis-bot
left a comment
Collaborator
There was a problem hiding this comment.
PR Review
Solid POC. The architecture is well-considered: decision-only step loop, host-owned no-downgrade ratchet, env-var-only credential handling, and thorough test coverage with a real loopback judge stub. The code is clean, well-documented, and the failure modes are carefully thought through.
Three findings below, all Medium.
Summary
| Severity | Count |
|---|---|
| Critical | 0 |
| Large | 0 |
| Medium | 3 |
[Medium] session_final() allocates a String via .to_ascii_lowercase() on every request carrying the x-switchyard-session-final header. Use eq_ignore_ascii_case for zero-allocation case-insensitive comparison. See inline comment.
[Medium] The x-switchyard-session-final header lets any client evict any session's floor if they can guess or know another session's ID, since session IDs are unauthenticated. The blast radius is bounded (layer 1 pass-through-on-failure still holds, so the worst case is re-judging, not forced downgrade), and the PR body + docs are honest about the floor's loss profile. Acceptable for a POC, but worth noting for production hardening: consider tying eviction to an authenticated identity, or relying solely on TTL eviction and dropping the client-facing final header.
[Medium] docs/identity-metadata.md (94 lines) defines an identity metadata contract for api_key_auth, metering, and budget filters. It is not mentioned in the PR description, is unrelated to switchyard_route, and references issues in a different repo (praxis-proxy/ai). Consider splitting it into its own PR or at minimum calling it out in the PR body so reviewers know it is intentionally included.
| .ok_or(RouteError::Body("is missing or empty"))?; | ||
| let value: serde_json::Value = serde_json::from_slice(raw).map_err(|err| RouteError::Json(err.to_string()))?; | ||
| if value.is_object() { | ||
| Ok(value) |
Collaborator
There was a problem hiding this comment.
[Medium] .to_ascii_lowercase() allocates a String on every request carrying this header. Use zero-allocation comparisons instead:
fn session_final(headers: &http::HeaderMap) -> bool {
headers
.get(SESSION_FINAL_HEADER)
.and_then(|value| value.to_str().ok())
.is_some_and(|text| {
let trimmed = text.trim();
trimmed.eq_ignore_ascii_case("true")
|| trimmed.eq_ignore_ascii_case("yes")
|| trimmed.eq_ignore_ascii_case("on")
|| trimmed == "1"
})
}|
|
||
| /// Reads the session id from the configured header, if present. | ||
| fn session_id(&self, ctx: &HttpFilterContext<'_>) -> Option<String> { | ||
| let value = ctx.request.headers.get(self.config.session_header.as_str())?; |
Collaborator
There was a problem hiding this comment.
[Medium] commit_floor evicts based on x-switchyard-session-final sent by the client, keyed by a client-supplied x-switchyard-session-id. Since session IDs are unauthenticated, a client who knows another session's ID can send both headers to force-evict the victim's floor, defeating the no-downgrade ratchet for that session's next turn.
For this POC this is acceptable (the docs are honest about the floor's loss profile, and layer 1 pass-through-on-failure still holds). For production hardening, consider tying eviction to an authenticated identity or dropping the client-facing eviction header entirely in favor of TTL-only expiry.
| @@ -0,0 +1,94 @@ | |||
| # Identity metadata contract | |||
Collaborator
There was a problem hiding this comment.
[Medium] This 94-line identity metadata contract is unrelated to switchyard_route and is not mentioned in the PR description. It defines filter_metadata keys for api_key_auth, metering, and budget filters, referencing issues in praxis-proxy/ai. Consider splitting it into its own PR, or at minimum calling it out in the PR body so reviewers know it is intentionally in scope.
Member
Author
|
I'm leaving this sitting for now. @yehuditkerido may take over this work -- which would be preferable since I think there are probably nuances that I haven't looked into yet. This approach was just about quickly reaching a place where we could offer classification based model picking. |
usize
added a commit
to usize/experimental
that referenced
this pull request
Aug 24, 2026
Turn this repo from an empty workspace into a home for experimental Praxis AI filters, and make it produce a runnable gateway image. - `praxis-experimental-filters`: carries the `[package.metadata.praxis-filters]` marker and registers via `praxis_filter::export_filters!`. Ships one no-op placeholder filter so discovery and registration are provable before real filters land. - `praxis-experimental-server`: thin bin whose `build.rs` runs praxis-ai's discovery (`praxis-ai-build-support`) and composes the stock praxis-ai server with this workspace's filters. praxis-ai is pinned to an exact rev because its crates are `publish = false`. - `Containerfile`: build and ship `praxis-experimental-server` rather than only the probe binary, following praxis-ai's cache-stub layout. The server's real `build.rs` is copied before the dependency-cache build; a stub would compile but emit no registration code, silently producing a server with none of this workspace's filters. - `deny.toml`: allow the praxis-ai git source. - `docs/identity-metadata.md`: the `filter_metadata` identity contract that identity producers write and metering/budget consumers read. Track A of the Standalone AI Gateway MVP epic (praxis-proxy/ai#758). Split out of praxis-proxy#4 so the image build does not depend on `switchyard_route`, which is being handed off separately. Assisted by Opus 5 Signed-off-by: usize <mofoster@redhat.com>
usize
added a commit
that referenced
this pull request
Aug 24, 2026
* feat: scaffold the experimental filters crate and gateway image Turn this repo from an empty workspace into a home for experimental Praxis AI filters, and make it produce a runnable gateway image. - `praxis-experimental-filters`: carries the `[package.metadata.praxis-filters]` marker and registers via `praxis_filter::export_filters!`. Ships one no-op placeholder filter so discovery and registration are provable before real filters land. - `praxis-experimental-server`: thin bin whose `build.rs` runs praxis-ai's discovery (`praxis-ai-build-support`) and composes the stock praxis-ai server with this workspace's filters. praxis-ai is pinned to an exact rev because its crates are `publish = false`. - `Containerfile`: build and ship `praxis-experimental-server` rather than only the probe binary, following praxis-ai's cache-stub layout. The server's real `build.rs` is copied before the dependency-cache build; a stub would compile but emit no registration code, silently producing a server with none of this workspace's filters. - `deny.toml`: allow the praxis-ai git source. - `docs/identity-metadata.md`: the `filter_metadata` identity contract that identity producers write and metering/budget consumers read. Track A of the Standalone AI Gateway MVP epic (praxis-proxy/ai#758). Split out of #4 so the image build does not depend on `switchyard_route`, which is being handed off separately. Assisted by Opus 5 Signed-off-by: usize <mofoster@redhat.com> * ci: smoke-test the gateway image instead of running a probe The container workflow ran `docker run --rm` and expected the process to exit, which was correct for the probe binary but hangs forever now that the entrypoint is a long-running server. Start the container detached with a minimal config, poll the Containerfile's HEALTHCHECK until it reports healthy, and confirm the gateway answers on :8080. Logs are dumped unconditionally so a failure is diagnosable. `examples/configs/minimal.yaml` is the smallest config that boots the server and answers health checks — no providers, no auth, no token accounting. It exists to prove the image runs; the quickstart config lands with the MVP packaging work. The admin listener binds loopback (core requires this unless `insecure_options.allow_public_admin` is set), so /healthy is reachable only from inside the container, which is where HEALTHCHECK runs. The proxy listener binds 0.0.0.0 so the published port is reachable from the runner. Verified locally: server boots against this config, /healthy returns {"status":"ok"}, / returns the expected body, and unknown paths 404. Assisted by Opus 5 Signed-off-by: usize <mofoster@redhat.com> * fix(ci): satisfy audit, coverage, and taplo gates Three CI gates failed on the first push: - **audit**: three advisories in the Pingora fork's tree, reached through praxis-ai and not under our control. Ignored with the same rationale praxis-proxy/praxis and praxis-proxy/ai already use for the first two; `lru` is newer and not yet in their lists. - **coverage**: 83% lines against a 90% floor. The placeholder's `from_config` was never exercised, only its registration. Added tests that build it from both a populated and an empty config mapping, bringing the workspace to 95% lines / 97% regions. - **meta-lint**: `taplo fmt --check` on the workspace `members` array and the new `deny.toml` ignore list. `HttpFilterContext` has no public constructor and praxis-filter's `test_utils` is `pub(crate)`, so an external filter crate cannot unit-test `on_request` directly. The placeholder's `on_request` is therefore left to integration coverage rather than faked here. Assisted by Opus 5 Signed-off-by: usize <mofoster@redhat.com> * docs: align identity-metadata table for MD060 meta-lint enforces MD060 table-column-style "aligned": every row's pipes must line up with the header separator. The separator row was narrower than the header, so the closing pipes drifted. Content unchanged; `git diff -w` is empty. This restores the alignment fix from experimental#4 that the scaffold split did not carry over. Verified: markdownlint-cli2@0.23.2 over all tracked *.md reports 0 issues. Assisted by Opus 5 Signed-off-by: usize <mofoster@redhat.com> * refactor(filters): follow test conventions in the placeholder module Two convention violations from review: - Test functions carried `///` doc comments. Per docs/conventions.md the function name is the documentation, so the comments are removed; the explanatory text was already duplicated in the assertion messages, which is where the conventions say it belongs. - The test module had no preceding separator. Added the full-width (77-dash) `Tests` separator, matching crates/experimental-probe. No behavior change; the same four tests pass and coverage is unchanged at 95% lines / 97% regions. Assisted by Opus 5 Signed-off-by: usize <mofoster@redhat.com> * docs: drop the identity metadata contract from the scaffold The doc proposed a cross-repo `identity.*` filter_metadata namespace, with normative rules ("exactly one producer per pipeline", "a later producer MUST NOT overwrite") and named producers and consumers owned by other repos (ai#698, ai#130, ai#577). Its own header said "Status: proposed... needs maintainer sign-off before consumers build on them". That is a proposal, not scaffolding, and it now has a home: the Discussion -> Proposal -> Experimental -> Standard process in praxis-proxy/enhancements. Merging it here as documentation would skip the sign-off it asks for, and would risk `api_key_auth` being written against a namespace that ai#698 later contradicts. Nothing in the scaffold reads it, so removing it is inert. It will be re-filed as a proposal, informed by a working `api_key_auth` prototype — the experimental-phase guide notes the How section is stronger that way. Narrows the scope of #1, which had listed the contract as a deliverable. Assisted by Opus 5 Signed-off-by: usize <mofoster@redhat.com> --------- Signed-off-by: usize <mofoster@redhat.com>
usize
marked this pull request as draft
August 24, 2026 17:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the
switchyard_routePOC filter (Capability-mode Mixture-of-Models routing) around NVIDIA NeMo Switchyardswitchyard-libsyv0.2.0.Part of #2.
What's here
The
switchyard-filtersandswitchyard-servercrates undercrates/(both workspace members), plus:switchyard-protocol.CallLlmvia Praxis'sSubRequestClientand reads the routingDecision, dropping the stream before the answer call.ctx.cluster+switchyard_route.*filter metadata (cluster / tier / model).on_failure); on failure the client's ownmodelis left untouched — the filter never causes a downgrade.floor.rs):max(floor, decision)ratchet, TTL-evicted, dropped onx-switchyard-session-final.judge.auth.value_env): the secret stays in the environment, only the variable name is in YAML; resolved once at startup, sent sensitive.session_floor.escalation_ratchetcost knob (default off): skip the judge callout once a session is flooredstrong, since the verdict is then foregone — the filter-side equivalent of Switchyard'sAffinityRouter::with_latch_only(["strong"]).docs/switchyard-route.md) and a runnable local demo (hack/switchyard-demo/).Verification
make allgreen: 54 filter tests + workspace tests, clippy strict wall, fmt, docs (warnings-denied), coverage ~93% lines. The demo is verified end-to-end against a real judge (openai/gpt-4ovia OpenRouter): easy→weak, hard→strong, the floor holds strong on a later easy turn, and the escalation ratchet skips the judge on turn 3 (three judge calls, not four).Related issues
Not in scope (per #2)
Proposal doc PR; async judge-based quality sampling;
model_rewritepipeline; durable/cross-replica session state (#3); building/shipping a classifier model; streaming judge responses, hedged answer calls,StageRouterandCustomclassifier modes.🤖 Generated with Claude Code