Skip to content

feat(client): add video_url content part for multimodal video input - #107

Open
octo-patch wants to merge 1 commit into
mlhher:mainfrom
octo-patch:octo/20260729-input-capability-recvq8lDDjeP4L
Open

feat(client): add video_url content part for multimodal video input#107
octo-patch wants to merge 1 commit into
mlhher:mainfrom
octo-patch:octo/20260729-input-capability-recvq8lDDjeP4L

Conversation

@octo-patch

Copy link
Copy Markdown

Reason: Add a video content-part type and request serialization so the OpenAI-compatible chat schema can express video input, preserving existing text/image handling.

Changes:

  • Add ContentPartVideoURL content-part type and a VideoURL payload struct (mirroring ImageURL) in internal/client/types.go, serialized as the OpenAI-compatible video_url field.
  • Wire video attachments into the orchestrator's Submit path alongside images: files detected as video/* are encoded as video_url content parts when the model supports vision; otherwise they fall back to a text note (parallel to the existing image behavior).
  • Allow the TUI file picker to attach video files using the same vision-capability gate as images, and keep media attachments consistent on model changes.

Checks:

  • go build ./...
  • go vet ./internal/client/ ./internal/orchestrator/ ./internal/tui/
  • go test ./internal/client/ ./internal/orchestrator/ ./internal/tui/

@mlhher

mlhher commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Thank you for your contribution. I see that you removed the PR template.
As thus I must ask you to manually confirm the CLA.

Please read the CLA here: Contributor License Agreement
If you agree, please reply to this thread with the following exact phrase:
I have read and agree to the terms of the Contributor License Agreement.

Will be tested and if nothing comes up can be merged afterwards. Thank you!

@giveen

giveen commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

I'm confused on what an IDE needs with video input.

@mlhher

mlhher commented Jul 29, 2026

Copy link
Copy Markdown
Owner

There are some code -> video workflows where this can come in handy, also generally I think it can be helpful to attach videos as workflows instead of manually explaining things. It is a minimal targeted change and should equally support Gemma, I am not against it.

@giveen

giveen commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Okay cool, I was just really curious on how this would work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants