fix(gemini): type mp4/mov/webm uploads so Gemini actually receives them

[mkubenka]

The bug

GEMINI_UPLOAD_MIME_TYPES covers only images and PDF. Everything else falls through to application/octet-stream, and Gemini silently discards uploads it cannot type.

The failure is invisible from the outside, which is what makes it expensive:

$ oracle --engine browser -m gemini-3.1-pro --browser-attachments always \
    --file clip.mp4 -p "Describe this video. If you did not receive a video, say NO VIDEO RECEIVED."

Answer:
NO VIDEO RECEIVED.

7.4s · gemini-3.1-pro[browser] · ↑51 ↓5
files=1        <-- reports the file as sent

The upload succeeds, the payload is built, files=1 is printed — and the model gets nothing. Reading the source reinforces the wrong conclusion, because the upload path genuinely is complete and correct. Only the MIME label is wrong.

Declaring video/mp4 makes the same file work on the first attempt:

Answer:
The video features an interview setup in a studio environment.
* **Visuals:** A man wearing glasses and a black shirt sits in front of a microphone...
  Chinese and English subtitles appear at the bottom of the screen...
### Verbatim English Subtitles (First 15 Seconds):
* "so it snows."

The change

Adds exactly three entries — .mp4, .mov, .webm — and extracts the lookup into an exported resolveGeminiUploadMimeType() so it is unit-testable. No behaviour change for currently-mapped types; the application/octet-stream fallback is preserved.

Why only three

I first wrote a much larger table from the Gemini API's documented formats, then probed each candidate against the live endpoint with a sentinel prompt separating "received" from "actually viewable". Most of it was wrong:

Format	Result
.mp4 .mov .webm	viewable — describes people, clothing, on-screen text
.avi	received but not viewable; only echoed container metadata (Lavf62.3.100)
.m4v	dropped, despite byte-identical content to a working .mp4 and the same declared video/mp4
.mkv .3gp .heic	Gemini returns an error
.wav .mp3 .m4a	never delivered — tried audio/mpeg, audio/mp3, audio/wav, audio/x-wav
.txt	never delivered — tried text/plain and text/md

Two conclusions worth recording:

1. Audio and plain text are not supported by this endpoint at all, whatever type is declared. Mapping them would advertise support that does not exist.
2. The endpoint gates on the file extension, not just the declared MIME type. The .m4v case is the proof: same bytes, same video/mp4, still dropped. So this table cannot be extrapolated from the Gemini API docs — each entry needs probing.

I have kept the entries to what I verified. If you have a broader whitelist from the web app, more can be added safely.

Verification

Against gemini-3.1-pro:

• 1.7 MB / 60 s MP4 — before: NO VIDEO RECEIVED; after: accurate scene description plus verbatim on-screen subtitles, cross-checked against frames read locally.
• A full 99-minute video in ten ~8.6 MB chunks (scale=640:-2,fps=8, crf 32). Answers to timestamp-specific questions matched an independently produced OCR transcript, including subtitles at 5:00 and 5:48 and both speakers' names.

Checks: vitest (5 new tests plus existing tests/gemini.test.ts — 15 passing), tsc --noEmit, oxfmt --check, oxlint.

Notes for the maintainer

• The 1 MB DEFAULT_MAX_FILE_SIZE_BYTES still applies, so video realistically needs --max-file-size-bytes / ORACLE_MAX_FILE_SIZE_BYTES. Left alone as a separate policy decision.
• uploadGeminiFile() reads the whole file into a Buffer→Blob, so large media will exhaust the heap. Chunking is the practical answer; out of scope here.
• The deeper issue is the silent failure, not the missing entries. An extension falling back to application/octet-stream arguably deserves a warning, since the current behaviour reports success while the model receives nothing — that is what made video look impossible rather than merely broken. There is no logger plumbed into client.ts's upload path, so I kept this PR to the data fix. Happy to follow up if you want the warning.

[mkubenka]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Review in progress
• Detail: The exact-review queue leased this run; Codex is reviewing the item.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/29943205924
• Updated: 2026-07-22T17:39:11.512Z

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 2, 2026, 1:36 PM ET / 17:36 UTC.

ClawSweeper review

What this changes

Maps MP4, MOV, and WebM Gemini browser uploads to video MIME types, extracts the lookup into a testable resolver, and adds focused regression coverage.

Merge readiness

⚠️ Ready for maintainer review - 1 item remains

This PR remains necessary: current main still assigns MP4, MOV, and WebM Gemini uploads the generic binary fallback. The refreshed branch is cleanly mergeable, has maintainer-confirmed live and built-library proof, and contains no actionable patch defect; keep it open for the owner-controlled landing path rather than closing it under the conservative repository profile.

Priority: P1
Reviewed head: 669819a945e5eee6d960b9cd16da63c8eaf67310

Review scores

Measure	Result	What it means
Overall readiness	🦞 diamond lobster (5/6)	Strong real behavior proof and a small, regression-tested repair make this a high-confidence landing candidate.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (live_output): The PR includes concrete before/after live Gemini output, and the owner’s follow-up reports focused tests plus a built-library MIME resolver smoke on the refreshed branch.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (live_output): The PR includes concrete before/after live Gemini output, and the owner’s follow-up reports focused tests plus a built-library MIME resolver smoke on the refreshed branch.
Evidence reviewed	5 items	Current-main defect: Current main maps only image and PDF extensions; the upload path falls back to application/octet-stream for .mp4, .mov, and .webm. Focused implementation and tests: The PR adds the three verified video mappings, routes upload typing through the resolver, and adds five resolver tests covering videos, existing mappings, case normalization, unsupported formats, and unknown extensions. Existing upload-path contract: The established upload test already verifies that the selected Blob MIME type is repeated in Gemini’s generation payload, so exercising the resolver directly targets the data that controls this behavior.
Findings	None	None.
Security	None	None.

How this fits together

Oracle’s Gemini browser adapter reads a local attachment, chooses its MIME type, uploads it to Gemini, and carries that type into the generation request. The mapping therefore determines whether an attached video is sent in a form the Gemini web endpoint can interpret.

flowchart LR
  A[CLI attachment path] --> B[Gemini browser adapter]
  B --> C[MIME type resolver]
  C --> D[Gemini upload request]
  D --> E[Generation payload]
  E --> F[Gemini model response]

Loading

Before merge

• Complete next step (P2) - No repair lane is needed: there are no actionable review findings, and merging remains an owner-controlled normal PR action.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Focused patch	3 files; 68 additions, 2 deletions	The change is contained to the upload-type mapping, its regression tests, and one unreleased release-note entry.
Verified mappings	3 new video extensions	The PR deliberately limits behavior change to MP4, MOV, and WebM formats reported as live-verified.

Technical review

Best possible solution:

Merge the narrow, endpoint-verified video MIME mappings with their regression tests while retaining the generic fallback for formats not proven to work with Gemini’s web upload endpoint.

Do we have a high-confidence way to reproduce the issue?

Yes, from source and supplied live evidence: current main sends these video extensions as application/octet-stream, while the PR body documents a before/after Gemini run using the same MP4 input.

Is this the best way to solve the issue?

Yes: an extension-to-MIME mapping is the narrowest maintainable repair because the existing upload and generation paths already propagate the selected MIME type correctly.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f5c087616b55.

Labels

Label justifications:

• P1: Supported video attachments are reported as sent but silently unavailable to Gemini, breaking a real browser attachment workflow.
• rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR includes concrete before/after live Gemini output, and the owner’s follow-up reports focused tests plus a built-library MIME resolver smoke on the refreshed branch.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR includes concrete before/after live Gemini output, and the owner’s follow-up reports focused tests plus a built-library MIME resolver smoke on the refreshed branch.

Evidence

What I checked:

• Current-main defect: Current main maps only image and PDF extensions; the upload path falls back to application/octet-stream for .mp4, .mov, and .webm. (src/gemini-web/client.ts:47, f5c087616b55)
• Focused implementation and tests: The PR adds the three verified video mappings, routes upload typing through the resolver, and adds five resolver tests covering videos, existing mappings, case normalization, unsupported formats, and unknown extensions. (tests/gemini-upload-mime.test.ts:5, 669819a945e5)
• Existing upload-path contract: The established upload test already verifies that the selected Blob MIME type is repeated in Gemini’s generation payload, so exercising the resolver directly targets the data that controls this behavior. (tests/gemini-web/upload.test.ts:17, f5c087616b55)
• Feature provenance: Gemini web uploads and the original MIME map were introduced with the browser-profile feature; the current missing mappings date to that initial implementation rather than a later regression. (src/gemini-web/client.ts:47, 64f83912f757)
• Maintainer integration and proof: A repository owner merged current main into the branch and reported focused resolver tests, format/typecheck/lint/build, a built-library MIME smoke, and no actionable autoreview finding; the supplied PR state reports the refreshed head as mergeable. (src/gemini-web/client.ts:47, 669819a945e5)

Likely related people:

• Edward Gao: Introduced the Gemini browser upload client and its original MIME table in the initial feature commit. (role: introduced behavior; confidence: high; commits: 64f83912f757; files: src/gemini-web/client.ts, tests/gemini-web/upload.test.ts)
• Peter Steinberger: Reconciled the release note and twice integrated current main into this PR branch, including the current mergeable head. (role: recent area contributor; confidence: high; commits: b7461a72b90d, c94c844efff7, 669819a945e5; files: CHANGELOG.md, src/gemini-web/client.ts)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (22 earlier review cycles; latest 8 shown)

• reviewed 2026-08-01T18:18:53.888Z sha 4ddc53c :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T19:46:52.771Z sha 4ddc53c :: needs changes before merge. :: [P2] Move the release note to the active unreleased section
• reviewed 2026-08-01T23:22:13.846Z sha 4ddc53c :: needs changes before merge. :: [P2] Move the Gemini release note to the active unreleased section
• reviewed 2026-08-02T05:29:52.293Z sha c94c844 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T10:20:30.886Z sha c94c844 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T12:20:34.966Z sha c94c844 :: needs changes before merge. :: [P2] Resolve the current changelog merge conflict
• reviewed 2026-08-02T13:56:16.629Z sha c94c844 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T16:19:24.127Z sha c94c844 :: needs maintainer review before merge. :: none

[steipete]

Maintainer repair pushed through c94c844e.

I reconciled the release note to the current 0.16.2 — Unreleased section, kept it to house style with contributor credit, and merged current main without rewriting Michal's original commit.

Proof on the resulting branch:

• 5 focused MIME resolver tests passed
• format, typecheck, lint, and build passed
• built-library smoke returned video/mp4, video/quicktime, and video/webm, while .m4v remained application/octet-stream
• independent Codex autoreview found no actionable defect

Recommendation: LAND once the refreshed checks complete.