Skip to content

media: harden ordered image, audio, MP3, and WebP handling - #23

Draft
pd95 wants to merge 5 commits into
feature/apertus-mlxfrom
feature/gemma4-mlx-multi-media
Draft

pd95 wants to merge 5 commits into
feature/apertus-mlxfrom
feature/gemma4-mlx-multi-media

Conversation

@pd95

@pd95 pd95 commented Aug 28, 2026

Copy link
Copy Markdown
Owner

Motivation

Media requests can expand substantially while being decoded or converted.
Without explicit bounds, one request can retain excessive decoded audio,
oversized WebP intermediates, or too many individually valid media items before
model execution begins. Format handling was also inconsistent between the CLI,
Gemma, MLX, and llama.cpp request paths.

These are shared request-boundary and interoperability concerns rather than a
single model's implementation details.

What this changes

  • adds MP3 decoding through a small miniaudio wrapper with explicit
    input/output limits and a clear no-CGO rejection path;
  • accepts MP3 attachments consistently through the CLI and Gemma 4 path;
  • bounds WebP dimensions and decoded allocation before conversion;
  • applies WebP normalization consistently to llama.cpp completion media;
  • enforces cumulative request-wide limits across ordered image and audio
    inputs, not only a limit on each attachment;
  • adds malformed and boundary coverage for decoders and request routes;
  • documents MP3 CLI support and the vendored decoder dependency.

Replacement, dependency, and scope

Official Ollama v0.34.0 contains the core Gemma 4 image/audio implementation
previously proposed in closed PR #22. This PR supersedes #22, is based on
Apertus PR #6, and carries only the remaining shared-media work. It preserves
upstream's bounded chunking and 600-second audio cap.

Compare:
feature/apertus-mlx...feature/gemma4-mlx-multi-media

Validation

CGO-enabled media, CLI, file-data, LLM, and Gemma tests pass, as does the
explicit no-CGO MP3 rejection path. Required Linux lint/build and serialized
Darwin MLX build gates also pass. The vendored miniaudio.h whitespace is the
only approved diff-hygiene exception.

Packaged ordered-media runtime validation remains part of the release gate.

Refresh and AI assistance

Adapted onto Ollama v0.34.0; head
c17a130842ad9555350e8c49a621776394771a37. Codex materially assisted
implementation review, tests, refresh validation, and this PR description.

@pd95

pd95 commented Aug 28, 2026

Copy link
Copy Markdown
Owner Author

This PR replaces the earlier v0.32.x review in #17 after the stack was refreshed onto the v0.33.1 release base. #17 remains the historical review record; this PR is the active v0.33.1 review.

@pd95
pd95 force-pushed the feature/gemma4-mlx-multi-media branch 2 times, most recently from 1b692a7 to e2ce337 Compare September 5, 2026 09:33
@pd95
pd95 changed the base branch from feature/gemma4-mlx-image-audio to feature/apertus-mlx September 5, 2026 09:42
@pd95
pd95 force-pushed the feature/gemma4-mlx-multi-media branch from e2ce337 to c17a130 Compare September 14, 2026 06:09
@pd95
pd95 removed this pull request from stack #26 September 14, 2026 18:24
@pd95
pd95 added this pull request to stack #27 September 14, 2026 18:24
pd95 and others added 5 commits September 18, 2026 08:30
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
@pd95
pd95 force-pushed the feature/gemma4-mlx-multi-media branch from c17a130 to 276e227 Compare September 18, 2026 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant