Conversation
llama.cpp build changes resulted in duplicate symbols between libllama and libmtmd. This moves the compat patch into libllama with exported symbols.
The runner package used to pick an engine from the first argument of the runner subcommand. Only the MLX engine is left, so the dispatcher has a single arm, its README still describes the removed Go runner's flags and endpoints, and the standalone cmd/runner binary exists only to invoke it. We call mlxrunner.Execute directly from the hidden runner subcommand and drop the --mlx-engine argument from the command line the MLX client spawns. Both sides ship in the same binary, so nothing has to accept both forms. The runner package and cmd/runner are removed. The subcommand's help hook hands the runner a bare --help. cobra calls the hook with no arguments for `ollama help runner`, which used to index past the end of the slice.
Several pieces outlived the code that used them. The root tokenizer package implemented the GGUF-side vocabularies for the Go engine and the safetensors-to-GGUF converter; nothing has imported it since the converter went. ml/backend.go held the Go engine's Backend, Context and Tensor interfaces, with fs.Config existing only to be returned from them, and a single CUDA template instance under ml/backend/ggml survived the engine removal along with the gitattributes entries for that tree and the CI change-filter globs for it and for the long-gone llama/llama.cpp. From the image generation engine, an integration test group that no test registers, its build tag, and the StepBar progress widget remained. DeviceInfo.IsBetter has no caller at all. All of it goes. Tidying the module file drops the regexp2 dependency and leaves protobuf as an indirect requirement. The llama3.2 tokenizer fixtures stay: the MLX runner's tokenizer uses them for its GGML parity test.
The MLX runner is the only Go inference runner left and is no longer experimental, so its packages leave x/. The bindings become a top-level mlx package beside the carried patches in mlx/compat, mirroring how llama/ holds the llama.cpp integration, and the runner becomes mlxrunner with the architectures nested under the package they implement. Subpackages move with their parent unless listed. x/mlxrunner/mlx mlx x/internal/mlxthread mlx/mlxthread x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest x/internal/mlxtest mlx/mlxtest x/quant mlx/quant mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx) x/mlxrunner mlxrunner x/models/nn mlxrunner/nn x/models/<arch> mlxrunner/model/<arch> x/mlxrunner/imports.go mlxrunner/model/architectures (new package) x/create create x/safetensors fs/safetensors x/tokenizer mlxrunner/tokenizer Every package keeps its name, so the Go changes are the import path rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift check and Darwin payload script follow the new paths. Four edits are not paths: the runner's blank architecture imports become the package mlxrunner/model/architectures, so the list to extend for a new model sits beside the architecture directories; a depguard rule keeps the two test harnesses out of non-test code, as the x/internal placement used to; the CI change filter's two entries for the long-deleted x/imagegen/mlx now name the bindings' CMake project and the carried patches, so a change to either builds the payload; and the tokenizer parity test reads its fixtures from its own testdata instead of walking out of x/. x/server and x/imagegen/manifest stay for the next two commits.
model/base held the Model interface and the architecture registry while model held weight loading and quant parameters, and every architecture imported both. create/client was the CLI side of safetensors imports, with the create command as its only caller, a duplicate of the command's adapter error, and a name that read like a second API client. The safetensors show helpers had their own package under x/ although it already declared package server. base merges into model, so base.Model and base.Register become model.Model and model.Register; nothing in the two overlapped. The create client's two files and their tests join package cmd, and the five names the command called are no longer exported. The show helpers join the server package, and the three entry points routes.go calls become unexported like the helpers around them.
The runner and its weight loader read a model's manifest through x/imagegen/manifest, the last piece of the removed image generation engine. It was a hand-rolled copy of the manifest package: its own model name parser with the default registry and namespace spelled out, its own blob path builder, and re-spelled media types, plus a model_index.json reader and other helpers that nothing has called since the engine went. The manifest package gains the three lookups the runner needs, a config layer by path, its contents, and the tensor layers, and ReadConfigJSON is built on the second of them. The weight loader resolves the model name with the shared parser, which fills in the same defaults the copy did, and locates blobs with BlobsPath. The architectures' calls to read their config.json compile unchanged. x/imagegen is gone.
nn.go held every layer type in the package apart from attention, recurrence and rope: the Linear and Embedding interfaces with their dense and quantized types, Conv1d, RMSNorm, LayerNorm and MultiLinear, with one test file to match. Finding a layer meant scanning the file named after the package. Each layer kind gets its own file: linear.go and embedding.go hold the interface and the dense and quantized types, conv.go, norm.go and multilinear.go take the rest, and nn_test.go splits the same way. The Layer and MultiLinearLayer interfaces go; nothing implemented or accepted them. No code changes otherwise.
model is one package with three jobs: the contract between the runner and the architectures, the opened checkpoint, and building nn layers from checkpoint tensors. Its files did not say which was which. base.go carried the folded package's name over the interfaces and the registry, root.go held the safetensors header scan next to Root, and quant.go mixed the nvfp4 global-scale helpers with quant parameter resolution. base.go becomes model.go, named for what it holds. root.go keeps Root and Open; TensorQuantInfo and the header scan join quant.go, so everything the checkpoint says about quantization is read and resolved in one file. The global-scale helpers move to globalscale.go with their tests. Root.Close, a no-op with one caller, goes. No code changes otherwise.
The decode loop releases MLX's pool of freed buffers every 256 generated tokens, which is also how often the KV cache grows and drops its previous, smaller buffers. The check fires only when the token count lands exactly on a multiple of 256. Speculative decoding emits several tokens per round, so most rounds step over the boundary and the pool is never released. Each growth at a long context leaves several GB of buffers that no later allocation can reuse, so the runner's footprint keeps climbing over a long generation until the system runs out of memory. We now release the pool whenever a round crosses a multiple of 256 tokens, which is what a single-token round already did. With qwen3.8:27b-mlx at a 98k-token context on a 128 GB machine, a long speculative generation previously grew the runner past 90 GB and panicked the kernel; it now stays flat at 30 GB.
…llama#18512) Redirects for registry and blob transfers now validate the target scheme and resolved addresses before following, re-check DNS on each redirect, and do not follow redirects that switch an https session to plain http. The --insecure option continues to relax address checks for private registries but not scheme checks.
Serialize final release jobs and delete old assets before retagging to avoid overlapping updates and mixed-version payloads. Keep uploads parallel and retry failures twice with 15s/30s delays. Retry cleanup using a fresh asset list and upload the checksum manifest only after all payloads succeed.
* nemotron_h: add MLX vision support Implement the RADIO vision encoder and projector on the shared MLX media pipeline, including dynamic-resolution preprocessing, deterministic placeholder expansion, chunked feature scattering, and MTP offsets. Expose source-advertised Nemotron vision while continuing to suppress unsupported audio, and preserve both modality towers at source precision during create. Harden Nemotron streaming parser termination and add focused coverage for vision configuration, media placement, capability reporting, and tool-call parsing. * review comments * address comments
Test both publication/deletion orderings directly instead of polling for a short-lived temporary file with a machine-dependent timeout.
…ntly (ollama#18438) getExistingName canonicalizes the case of each model name part (host, namespace, model, tag) by searching all manifests for a case-insensitive match. The original implementation matched each part independently — the tag from any manifest whose tag case-insensitively matched the requested tag would overwrite the tag, regardless of whether the host, namespace, or model matched. A 'set' variable was intended to track which parts had already been canonicalized and prevent overwrites, but it was never written to, so it was always zero-valued and every match overwrote the corresponding part unconditionally. With 3000+ manifests, if another model had a tag that case-insensitively matched (e.g. 'Q4_K_M' for a different model), the requested model's tag could be canonicalized to that other model's tag casing. Go's map iteration order is randomized, so the last match wins — producing intermittent 'model not found' errors that succeed on retry. Fix: when all four parts of an entry case-insensitively match the input, return that entry's canonical name directly. Otherwise canonicalize each part independently, with the 'set' variable now properly updated after each part is set so it is only written once. This handles both exact matches and new tags on existing models.
A format on a thinking model has to leave the thinking free and constrain only the content after it, so whatever enforces the format needs to know where the thinking ends. Today the server guesses whether a parser's response starts inside thinking from the think value alone, which is wrong for parsers whose default differs, and it has no way to learn the closing string at all. Each parser now answers ThinkingClose after Init: the strings any of which ends the thinking its response begins with, or none when the response starts in content because thinking is off, an assistant prefill continues content, or the parser suppresses thinking for tools. Parsers whose models open a new message before content end the thinking at that message's header. Nothing consumes the answer yet.
The MLX runner applies a format's grammar from the first sampled token, so a thinking model asked for a format cannot think first, and the server has to run two generations to get both the thinking and the formatted content. A completion request now carries the strings that end the thinking its response begins with, and the MLX client builds from them a structural tag: free text that cannot contain any of them, then one of them, then the schema. The tail is optional so a response may still end inside its thinking, as an unconstrained one can. Without a closing string the tag is the plain schema, as before. The server does not send the strings yet.
llama-server applies a schema from the first sampled token, so a thinking model asked for a format cannot think first, and the server has to run two generations to get both the thinking and the formatted content. The client now sends one request whose grammar leaves the text before a closing string unconstrained and requires the format after it. llama-server converts the schema for us: an empty completion evaluates and generates nothing but reports the GBNF it derived, which we wrap in rules that recognize the closing strings and cache per schema for the life of the process. On qwen3 0.6b at temperature 0 the thinking is byte-identical with and without a format and the JSON follows the schema. A response that ends before a closing string is delivered unchanged. The conversion request briefly takes a llama-server slot on a cache miss. The server does not send the strings yet.
A format on a thinking model ran two generations: an unconstrained one, cancelled once the parser reported content, then a re-rendered prompt with the parsed thinking under the grammar. The restart cost a second prefill, dropped the chunk that crossed the boundary, needed a harmony prompt hack, stitched metrics across the two requests, and on MLX could leak a stray first token into the JSON. The generate endpoint never deferred at all, so its JSON was forced inside the thinking. Both handlers now make one completion request that names the strings ending the response's thinking, from the builtin parser or the generic thinking parser, and the runner constrains only the content after them in a single generation. The prompt is evaluated once and metrics pass straight through. A raw generate prompt names no strings, since nothing says where its response starts, and its format applies from the first token as before. A format now applies to whatever follows the thinking, so a tool call can no longer take the place of formatted content, which was already the case with thinking off; harmony is the exception, since its tool calls precede the final message. The per-token metrics flag both runners carried for the cancelled first pass has no caller left and goes with the two-pass code and its tests. Fixes ollama#18441 Fixes ollama#17544 Fixes ollama#14196 Fixes ollama#10929
Refine memory allocation failure log substrings for upstream changes. Remove the no longer needed Laguna metal patch - fixed upstream.
Plumbs fast::gated_delta_update through a temporary MLX-C patch for now.
Replace the fixed checkpoint image budget with per-image selection across the supported 70, 140, 280, 560, and 1120 budgets. Choose the publisher resize grid closest to the input resolution, accounting for aspect ratio. This preserves more detail in high-resolution documents while allowing smaller images to use fewer tokens, without adding an API parameter. Cover budget boundaries, extreme dimensions, position limits, and media expansion for both vision architectures.
* mlx: speed up Qwen 3.8 prompt processing Use MLX's gated-delta kernel for long scans and fold dense MLP global scales into SwiGLU. * address comments
Move the ollama_xgrammar target into mlxrunner/xgrammar/native so it can be configured on its own against an installed xgrammar. cmake/mlx now adds it as a subdirectory and still uses the pinned xgrammar.
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
pd95
force-pushed
the
feature/responses-namespace-tools-upstream
branch
from
September 23, 2026 18:30
3a78148 to
6e8daeb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.34.1 refresh layer at 3a78148. Preserves injective namespace and tool-name identity across streaming, non-streaming, history, and tool search; focused and integrated gates passed.