feat(switchyard_route): add MoM routing filter POC - #16
Conversation
usize
left a comment
usize
left a comment
There was a problem hiding this comment.
Since this is experimental, I'm OK landing this as-is, after an automated review for test gaps and other nits.
Importantly, it would be great to see a follow-up that shows us what happens if Switchyard dies mid-session. What we want to avoid is thrashing between a "Strong" and "Weak" model.
311f4b3 to
dd7b7f9
Compare
|
PR too large: 1091 lines added (limit: 750, excludes Cargo files, tests, docs, examples, and benchmarks). Please split into smaller PRs. Add |
|
Non-conforming commit subjects (expected
Amend with |
90cf365 to
02d3cc1
Compare
Wire NVIDIA Switchyard Capability-mode classification into Praxis as a decision-only weak/strong router, with a local mock demo so reviewers can verify the plumbing without a cluster. Signed-off-by: Yehudit Kerido <ykerido@redhat.com>
02d3cc1 to
d693aa5
Compare
|
This PR is intentionally over the size limit (filter + demo together). Follow-ups I’ll open separately:
Happy to split further if reviewers prefer the demo in its own PR. |
praxis-bot
left a comment
praxis-bot
left a comment
There was a problem hiding this comment.
PR Review
Summary: Clean POC with well-structured config parsing, good judge retry logic, and appropriate error types. Three findings below -- one about fail-open semantics, two about edge-case handling.
| Severity | Count |
|---|---|
| Critical | 0 |
| Large | 1 |
| Medium | 2 |
| FailureMode::Open => { | ||
| // Demo: `run-demo.sh` greps `switchyard_route: fail-open`. | ||
| debug!("switchyard_route: fail-open, passing through"); | ||
| Ok(FilterAction::Continue) |
There was a problem hiding this comment.
[Large] Fail-open returns Continue but no cluster has been set in metadata, while selects_cluster() returns true (line 292). The downstream load_balancer will receive ctx.cluster = None and either error or pick an arbitrary cluster -- neither is "pass through unchanged." Either set a documented fallback cluster here (e.g. ctx.set_metadata(METADATA_CLUSTER, self.config.weak.cluster.clone()) with a comment explaining the default), or make selects_cluster() return false when on_failure is Open so the pipeline does not expect this filter to have selected one.
| pub(crate) fn parse(yaml: &serde_yaml::Value) -> Result<RouteConfig, FilterError> { | ||
| let raw: RawConfig = serde_yaml::from_value(yaml.clone()).map_err(|err| FilterError::from(err.to_string()))?; | ||
|
|
||
| if !(0.0..=1.0).contains(&raw.threshold) { |
There was a problem hiding this comment.
[Medium] (0.0..=1.0).contains(&f64::NAN) returns false, so !false passes NaN through validation. YAML's .nan literal deserializes to f64::NAN via serde, so threshold: .nan in a config file would silently poison every Switchyard comparison. Add || raw.threshold.is_nan() to the guard, or switch to raw.threshold.is_finite() && (0.0..=1.0).contains(&raw.threshold).
|
|
||
| /// Truncates a body for log-safe error previews. | ||
| fn body_preview(body: &Bytes) -> String { | ||
| String::from_utf8_lossy(body).chars().take(200).collect() |
There was a problem hiding this comment.
[Medium] String::from_utf8_lossy(body) allocates (or borrows) the full body -- up to DEFAULT_MAX_BODY_BYTES (1 MiB) -- before .chars().take(200) truncates it. On error paths with large payloads this is a needless allocation. Slice first:
let cap = body.len().min(800);
String::from_utf8_lossy(&body[..cap]).chars().take(200).collect()(800 bytes is enough for 200 chars even with 4-byte UTF-8 sequences.)
3 participants
Wire NVIDIA Switchyard Capability-mode classification into Praxis as a decision-only weak/strong router, with a local mock demo so reviewers can verify the plumbing without a cluster.
What?
switchyard_routeHTTP filter (crates/praxis-experimental-filters): buffer OpenAI chat body, run Switchyard Capabilityrun_stream, serve judgeCallLlmviaSubRequestClient, mapweak/strongto configured(cluster, model), rewrite bodymodel, setctx.clusterforload_balancer.config.rs):judge,threshold,targets.weak|strong,on_failure(open/closed); secrets only viajudge.auth.value_env.lib.rs; bump toolchain1.96.0→1.96.1(Switchyard 0.2.0); updateCargo.lock.demos/switchyard-route/): judge + weak/strong echo upstreams,run-demo.sh, template Praxis YAML — no cluster required.docs/switchyard-route.md) and link the demo fromdemos/README.md.Why?
Mixture-of-Models gateways need a clear place to decide which model tier should handle a request before it hits an upstream. Without that, every request either always pays for a strong model or always risks under-serving hard work on a weak one.
This PR shows that path in Praxis using NVIDIA Switchyard’s Capability classifier: classify the task, pick weak or strong, rewrite the request, and select the cluster—decision-only, with a local mock demo so the flow is easy to verify.
Testing
cargo test -p praxis-experimental-filters(registry + config parse)cd demos/switchyard-route && ./run-demo.sh— expect 3×served_by=weak-upstream, then 3×served_by=strong-upstream, plusswitchyard_route: judge verdict/routedin gateway logsNote: behavioral coverage for fail-open/closed, bad path, judge timeout, etc. is intentionally thin for this POC; demo is the main E2E check.
Related Issues
Relates to #2