Originating discussion: https://github.com/orgs/praxis-proxy/discussions/976
(maintainer +1: https://github.com/orgs/praxis-proxy/discussions/976#discussioncomment-18019663)
Parent epic: praxis-proxy/ai#758 (Track A; MVP success criterion 7)
Depends on: #1 (scaffold)
Summary
Build a POC switchyard_route HTTP filter in this repo around NVIDIA NeMo
Switchyard's switchyard-libsy (pre-alpha; pinned to v0.2.0. The filter buffers the request body
No classifier model needs to be shipped (the judge is an ordinary endpoint callout). We should think through failure modes carefully,
for example, what happens if we can no longer reach the classifier model? We should try to avoid turn by turn thrashing by e.g., having a fallback to a sensible model in place.
Acceptance criteria
- The composed experimental server routes
/v1/chat/completions traffic across two
models per the configured algorithm, with the judge served by callout —
demonstrable with the repo's kind tooling or a documented local recipe.
make all (build, fmt, lint, test, audit) passes.- Decision observable via
switchyard_route.cluster / .tier / .algorithm
filter metadata.
Out of scope
- Async judge-based quality sampling (@yossiovadia's complementary proposal —
awaiting its own discussion).
model_rewrite and the classify→validate→select→rewrite pipeline (praxis#838, ai#643).- Durable/cross-replica session state (upstream Switchyard has no storage hook in v0.2.0).
- Building or shipping our own classifier model (epic ai#74 / ai#715 territory).
- Streaming judge responses, hedged multi-candidate answer calls,
StageRouter
and Custom classifier modes (documented as follow-ups if the POC graduates).
Originating discussion: https://github.com/orgs/praxis-proxy/discussions/976
(maintainer +1: https://github.com/orgs/praxis-proxy/discussions/976#discussioncomment-18019663)
Parent epic: praxis-proxy/ai#758 (Track A; MVP success criterion 7)
Depends on: #1 (scaffold)
Summary
Build a POC
switchyard_routeHTTP filter in this repo around NVIDIA NeMoSwitchyard's
switchyard-libsy(pre-alpha; pinned to v0.2.0. The filter buffers the request bodyNo classifier model needs to be shipped (the judge is an ordinary endpoint callout). We should think through failure modes carefully,
for example, what happens if we can no longer reach the classifier model? We should try to avoid turn by turn thrashing by e.g., having a fallback to a sensible model in place.
Acceptance criteria
/v1/chat/completionstraffic across twomodels per the configured algorithm, with the judge served by callout —
demonstrable with the repo's kind tooling or a documented local recipe.
make all(build, fmt, lint, test, audit) passes.switchyard_route.cluster/.tier/.algorithmfilter metadata.
Out of scope
awaiting its own discussion).
model_rewriteand the classify→validate→select→rewrite pipeline (praxis#838, ai#643).StageRouterand
Customclassifier modes (documented as follow-ups if the POC graduates).