feat: add XGrammar structured output with speculative decoding

README.md

@@ -7,6 +7,9 @@ single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a
 OpenAI-/Anthropic-compatible HTTP APIs. The runtime is deliberately specialized: one GPU, one
 resident model, and a startup-fixed capacity of one to eight active requests.
 
+JSON object and JSON Schema constrained generation use vendored XGrammar v0.2.7. See
+[structured output](docs/serving.md#structured-output) for API examples, schema support, and limits.
+
 Five official artifacts are available. The quick-start commands use Qwen3.8-27B NVFP4.
 
 | Model | Weights | Artifact | Download and model card |

apps/cli/main.cpp

@@ -257,9 +257,11 @@ int main(int argc, char** argv) {
                 : ninfer::product::prompt_from_messages(cli.messages_path, cli.enable_thinking,
                                                         cli.enable_vision);
         input.options.reasoning_effort = cli.reasoning_effort;
+        ninfer::product::apply_structured_output_instruction(input, cli.structured_output);
 
         ninfer::RequestOptions request;
         request.execution.sampling                = cli.sampling;
+        request.execution.structured_output       = cli.structured_output;
         request.execution.requested_output_tokens = cli.max_new;
         request.execution.thinking.budget         = cli.thinking_budget;
         request.stop.token_ids                    = cli.stop_token_ids;

apps/cli/options.cpp

@@ -5,6 +5,8 @@
 #include <cmath>
 #include <cstdlib>
 #include <limits>
+#include <fstream>
+#include <iterator>
 #include <stdexcept>
 #include <string_view>
 
@@ -100,6 +102,8 @@ std::string usage_text(const char* argv0) {
            "Structured message content accepts text, image/image_url, and video/video_url parts;\n"
            "media sources may be local paths, HTTP(S) URLs, or base64 data URIs.\n"
            "--vision enables image/video input and loads the fixed Vision GPU allocations.\n"
+           "--json constrains output to a JSON object; --json-schema FILE enforces a supported "
+           "JSON schema. Thinking defaults off; an explicit effort or budget enables it.\n"
            "--thinking-budget caps model-origin thinking tokens; inserted control tokens count "
            "toward --max-new.\n"
            "--kv-capacity auto leaves " +
@@ -151,6 +155,17 @@ Options parse_options(int argc, char** argv) {
             options.speculative.draft_tokens = parse_u32(value(arg), "draft-tokens");
         } else if (arg == "--lm-head-draft") {
             options.speculative.proposal_head = ProposalHead::Optimized;
+        } else if (arg == "--json" || arg == "--json-schema") {
+            if (options.structured_output.kind != StructuredOutputKind::None) {
+                throw std::invalid_argument("choose exactly one structured output mode");
+            }
+            options.structured_output.kind = arg == "--json" ? StructuredOutputKind::JsonObject
+                                                             : StructuredOutputKind::JsonSchema;
+            if (arg == "--json-schema") {
+                std::ifstream schema(value(arg));
+                if (!schema) { throw std::invalid_argument("cannot read JSON schema file"); }
+                options.structured_output.schema.assign(std::istreambuf_iterator<char>(schema), {});
+            }
         } else if (arg == "--raw-output") {
             options.raw_output = true;
         } else if (arg == "--print-token-ids") {
@@ -224,6 +239,16 @@ Options parse_options(int argc, char** argv) {
         throw std::invalid_argument("--kv-capacity must be at least --max-context");
     }
     product::validate_speculative_cli_options(options.speculative);
+    if (options.structured_output.kind != StructuredOutputKind::None) {
+        if (options.raw_output || !options.stop_strings.empty() ||
+            !options.stop_token_ids.empty()) {
+            throw std::invalid_argument(
+                "structured output requires decoded text and default stops");
+        }
+        if (!options.reasoning_effort && !options.thinking_budget) {
+            options.enable_thinking = false;
+        }
+    }
     if (options.enable_thinking == false && options.reasoning_effort &&
         *options.reasoning_effort != ReasoningEffort::None) {
         throw std::invalid_argument("--reasoning-effort cannot be combined with --no-thinking");

apps/cli/options.h

@@ -41,6 +41,7 @@ struct Options {
 
     // Omitted fields are resolved from the loaded model and rendered prompt mode by Engine.
     SamplingOverrides sampling;
+    StructuredOutputOptions structured_output;
     bool greedy                 = false;
     product::LogLevel log_level = product::LogLevel::Info;
 };

cmake/Dependencies.cmake

@@ -11,6 +11,7 @@ target_include_directories(ninfer::json INTERFACE
 
 # Source base for the custom-template frontend; consumers will link it explicitly.
 add_subdirectory(third_party/llama-jinja EXCLUDE_FROM_ALL)
+add_subdirectory(third_party/xgrammar EXCLUDE_FROM_ALL)
 
 if(NINFER_BUILD_PRODUCT_SUPPORT)
   # Media acquisition uses CURLOPT_PROTOCOLS_STR and CURLOPT_REDIR_PROTOCOLS_STR,

docs/cli.md

@@ -280,3 +280,19 @@ from this one-request interface; the persistent Engine and server routes own cro
 optional Host backing.
 
 All weight, sequence, workspace, and graph allocations are released when the Engine is destroyed.
+
+## JSON and JSON Schema output
+
+`--json` constrains output to a JSON object. `--json-schema FILE` constrains it to the supported
+JSON Schema subset described in [serving](serving.md#structured-output). Thinking defaults off;
+an explicit `--reasoning-effort` or `--thinking-budget` retains reasoning before the constrained answer.
+The two format flags are mutually exclusive and reject raw output and custom stops. The CLI adds
+the requested format/schema to the model's instructions before tokenization. They work with
+ordinary, MTP, DFlash and DFlash2 execution.
+
+```bash
+./build/apps/ninfer model.ninfer --prompt 'Return the city as JSON' --json --max-new 128
+./build/apps/ninfer model.ninfer --prompt 'Return a weather record' --json-schema weather.schema.json --max-new 128
+```
+
+Read the reported finish reason: an output/context limit or cancellation can truncate the JSON.

docs/maintainer/engine-architecture.md

@@ -601,3 +601,41 @@ fixed-shape 和 device-specialized 实现）都归 `src/ops`。
   和 consumer address contract；
 - [Op development](op-development.md)：Op 正确性与性能准入；
 - [CLI](../cli.md)与 [HTTP serving](../serving.md)：外部行为。
+
+## Structured output transaction
+
+`StructuredOutputOptions` crosses the public Engine boundary as schema data. Frontend compiles it
+with a tokenizer-specific XGrammar compiler (bounded 256 MiB cache) and creates a request-owned
+`GrammarState`. Engine passes the same state through the base plan, admission plan, and request
+control. Sequence/checkpoint/cache state never owns it. OutputSession previews on a fork and
+moves the fork into the committed state only after Program commit succeeds. Cancellation before
+output preview advances no grammar state.
+
+CLI and serving use the shared product prompt adapter to expose an explicit response format and
+schema as a leading instruction before preparation. It retains caller message contents and remaps
+explicit cache boundaries if a system message is inserted. This aligns model behavior with the
+constraint, especially when an automatic tool-call alternative remains available; enforcement
+still belongs to the matcher. Requests without a format are unchanged.
+
+The model frontend supplies its reasoning-close delimiter and native tool envelope. The text layer
+composes these with the final JSON grammar into one matcher. Thus a candidate token, or a speculative
+block, may cross from reasoning into tool calls or final JSON without an out-of-band phase switch.
+The first reasoning-close delimiter ends unrestricted reasoning. The model's tool grammar uses
+declared names and optional non-strict parameters in prompt declaration order. OutputSession also previews and
+commits forced thinking-control tokens through the matcher; the injected closure cannot leave the
+grammar in the reasoning phase. Framing and grammar state have identical rollback/commit ownership.
+
+Program reserves vocabulary bitsets in its planned persistent device arena and owns pinned host
+staging. Ordinary/prefill sampling uses the current mask. MTP supplies each current draft prefix
+to a forked matcher before launch. DFlash/DFlash2 must first generate device drafts: their graph
+captures D2H draft transfer, a CUDA host function that fills all reachable prefix masks, H2D mask
+transfer, then target verification. The host function calls no CUDA API; it captures exceptions
+for rethrow after stream synchronization and before any token can be published. All node addresses
+belong to Program and outlive graph replay.
+
+SamplingConfig carries an optional bitset pointer and column stride. Column i describes the
+grammar after drafts[0..i). The mask is applied before penalties and sampling filters. After an
+illegal draft or EOS, suffix columns are unreachable and need no grammar traversal. Every reachable
+mask must be nonempty. Raw-logit fast paths require a null mask. Draft proposal probabilities are
+unchanged; target p is constrained and normalized before the existing p/q and residual calculation.
+Existing KV, recurrence replay, token-count, and terminal-prefix commit rules remain authoritative.