feat(routing): support json_object output for custom classifiers

[chethanuk]

What

Custom-mode llm_classifier routes can now set response_format_type = "json_object". On main the runner rejects that combination at config load. Capability and escalation routes already support it.

It works the same way as for the packaged classifiers. Switchyard appends the configured response_schema to the judge prompt, sends {"type": "json_object"}, and checks the verdict against the schema locally. A verdict that fails the check falls back to default_target. JSON Schema stays the default, so existing routes don't change.

• ClassifierContract::from_inner_schema takes the response format. from_config and the custom path share one helper, schema_in_prompt.
• In JSON Object mode, a response_schema whose root type doesn't allow object (a string other than "object", or an array without it) is rejected at load. The provider only returns objects, so such a schema would send every turn to default_target. A schema with no type at all is not checked.
• CustomClassifierConfig gets a response_format_type field (default JsonSchema). The runner passes it for the top-level route in build_algorithm and for the nested subagents classifier in build_subagent_router_config.
• The Python CustomClassifierConfig takes response_format_type="json_schema", parsed the same way as the other classifier configs. The switchyard_rust/libsy.py stub is updated.
• Docs: the response_format_type row in toml_schema.md and the custom section of llm_classifier_routing.md.

Why

Closes #429. Some providers support JSON Object mode but not JSON Schema, and custom classifiers couldn't use them. The contract follows the issue's conservative option: response_schema is still required in both modes and is the source of truth, so a prompt never needs its own copy. Provider wrappers such as {"json_schema": {...}} are still rejected as an inner schema.

Before / After

Same config and request against a stub OpenAI upstream. The config (custom-json-object.toml) and stub (mock.py) are in pr-evidence/6. Both binaries were built with cargo build --locked -p switchyard-server --bin switchyard-server, the stub was started with python3 mock.py 18431, and every output line below is copied verbatim from that run.

$ grep -E '^(mode|response_format_type) ' custom-json-object.toml
mode = "custom"
response_format_type = "json_object"

Before, base a601a9a3f9db149a1ad430fa43b1463a170c8a82 (fork main, before it was synced to upstream). The server exits with status 1:

$ switchyard-server --config custom-json-object.toml --host 127.0.0.1 --port 18432
invalid server config custom-json-object.toml: llm_classifier route custom mode custom cannot use capability or escalation fields and response_format_type must be 'json_schema': llm_classifier route custom mode custom cannot use capability or escalation fields and response_format_type must be 'json_schema': llm_classifier route custom mode custom cannot use capability or escalation fields and response_format_type must be 'json_schema'

After, PR head d9c819f599322108d8debd0df16608a7e7cb3547 (re-run at this head):

$ switchyard-server --config custom-json-object.toml --host 127.0.0.1 --port 18432 &
$ curl -s http://127.0.0.1:18432/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"switchyard/custom","messages":[{"role":"user","content":"Refactor the scheduler."}]}' | jq -r '.model, .choices[0].message.content'
strong/model
hello from strong/model
$ cat mock.log   # what the stub upstream received
[mock] judge response_format = {"type": "json_object"}
[mock] schema in judge prompt: True
[mock] completion served by strong/model

Notes for reviewers

Start with crates/libsy/src/algorithms/util/classifier_contract.rs. Both constructors run validate_prompt on the template, never on the prompt with the schema appended. The appended text is never empty, and a schema may itself contain the literal {{RESPONSE_SCHEMA}}.

This adds a public field to CustomClassifierConfig, so Rust callers that build it with a struct literal instead of new() must add response_format_type. It follows how other public config fields were added before 1.0.

build_subagent_router_config is easy to miss. Without the field there, a nested custom classifier would quietly stay on JSON Schema; subagent_custom_classifier_can_request_json_object_output covers it.

The server test mock used to recognize custom judge calls only by response_format.json_schema. It now also recognizes a json_object request whose prompt contains the custom schema.

Verification, at the head of this PR rebased on fbabf51c (upstream main):

cargo fmt --all --check                                                                      # clean
cargo clippy -p switchyard-libsy -p switchyard-runner -p switchyard-server -p switchyard-py \
  --all-targets -- -D warnings                                                               # clean
cargo test -p switchyard-libsy -p switchyard-runner -p switchyard-server                     # all pass (330 libsy unit tests, 64 runner, 58 server)

The new runner and server tests fail on main with the old config error. The Python tests (tests/test_libsy_minimal_bindings.py) pass locally against an extension built with maturin develop (22 passed, including the two new custom-classifier tests). They have not run on the fork's CI.

Summary by CodeRabbit

• New Features
  • Custom classifiers now support both JSON Schema and JSON Object response formats. JSON Schema remains the default.
  • JSON Object responses are checked against the configured response schema; invalid responses fall back to the default target.
  • Python configuration accepts response_format_type values of json_schema or json_object.
• Documentation
  • Updated classifier guidance to describe response format options and validation behavior.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA-NeMo/Switchyard/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2d402fb7-2264-4fbb-a082-e57803b4e84f

📥 Commits

Reviewing files that changed from the base of the PR and between fbabf51 and d9c819f.

📒 Files selected for processing (10)

• crates/libsy/src/algorithms/llm_class.rs
• crates/libsy/src/algorithms/util/classifier_contract.rs
• crates/switchyard-py/src/libsy_bindings.rs
• crates/switchyard-runner/src/algorithm.rs
• crates/switchyard-runner/src/config.rs
• crates/switchyard-server/tests/server.rs
• docs/reference/toml_schema.md
• docs/routing_algorithms/llm_classifier_routing.md
• switchyard_rust/libsy.py
• tests/test_libsy_minimal_bindings.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

Custom classifiers now support JSON Object response formatting as an alternative to JSON Schema mode. The configured schema is appended to the prompt in JSON Object mode, and returned verdicts are validated against it. The setting is available through runtime configuration and Python bindings.

Changes

Custom classifier response mode

Layer / File(s)	Summary
Define the response contract crates/libsy/src/algorithms/llm_class.rs, crates/libsy/src/algorithms/util/classifier_contract.rs	Custom classifier configuration stores a response format. Contract construction sends the schema through the JSON Schema wrapper or appends it to the prompt for JSON Object mode. Both modes validate verdicts locally.
Expose and propagate response format crates/switchyard-py/src/libsy_bindings.rs, switchyard_rust/libsy.py, crates/switchyard-runner/src/algorithm.rs, crates/switchyard-runner/src/config.rs	Python and runner configuration accept json_schema or json_object. The runner passes the selected format to parent and subagent custom classifiers.
Verify and document JSON Object routing crates/switchyard-server/tests/server.rs, tests/test_libsy_minimal_bindings.py, docs/reference/toml_schema.md, docs/routing_algorithms/llm_classifier_routing.md	Tests cover JSON Object requests, prompt contents, routing, and fallback when verdict validation fails. Documentation describes schema handling and the default_target fallback.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to d9c81

This change adds an opt-in JSON Object response mode for custom classifiers and leaves JSON Schema as the default. The supplied context shows no merge-blocking risk. The author notes the Python tests have not yet run on the fork's CI.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 54.05% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 37 functions across 8 files. (2 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: adding json_object output support for custom classifiers.
Linked Issues check	✅ Passed	The PR satisfies the coding requirements in [#429]. from_inner_schema requires and compiles an inner response_schema, rejects response-format wrappers, appends the configured schema in `json_objec…
Out of Scope Changes check	✅ Passed	The changed libsy contract, runner propagation, server integration tests, bindings, documentation, and minimal binding tests directly support [#429]. The top-level and nested subagents coverage veri…

Full details: Docstring Coverage

Explanation

Docstring coverage is 54.05% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 37 functions across 8 files. (2 skipped: 2 unsupported.)

• Fix all pre-merge checks with AI

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

_{A rabbit checks the schema in the moonlit glow
“JSON Object,” it whispers, “off we go!”
The prompt gets its schema, neat and clear
Each verdict meets a local check here
Fast or default, the routes appear
Then hop-hop-hop, the tests cheer!}

_{Comment @coderabbitai help to get the list of available commands.}