Add opt-in TwelveLabs Pegasus connector for chat-with-clip - #6
Open
mohit-twelvelabs wants to merge 1 commit into
Open
Add opt-in TwelveLabs Pegasus connector for chat-with-clip#6mohit-twelvelabs wants to merge 1 commit into
mohit-twelvelabs wants to merge 1 commit into
Conversation
Adds an opt-in TwelveLabsConnector that chats with a whole video clip using the Pegasus video understanding model, as an alternative to single-frame OpenAI inference. Includes a gradio example, README docs, and tests. Default OpenAI behavior is unchanged.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
This PR adds an opt-in way to chat with a whole webcam clip using the TwelveLabs Pegasus video understanding model, alongside the existing OpenAI single-frame path.
What it adds
webcamgpt.TwelveLabsConnector— mirrors the existingOpanAIConnectorAPI (simple_prompt(...)), but takes a video clip (local file or public URL) instead of a single frame. It accepts a local path (sent as base64) or anhttp(s)URL.webcamgpt.utils.encode_video_to_base64helper, in the same style asencode_image_to_base64.examples/webcam_twelvelabs.py— a gradio demo mirroringexamples/webcam.py, but recording a short clip and asking Pegasus about it.twelvelabs>=1.2.8added torequirements.txt; README section documenting the option.test/test_twelvelabs_connector.py— no-network unit tests for input routing + key validation, plus a live Pegasus test that is skipped unlessTWELVELABS_API_KEYis set.Why it helps this project
The webcam stream is inherently temporal — a single frame loses motion and actions. Pegasus reasons over the whole clip server-side, so users can ask "what just happened?" rather than "what's in this frame?".
Opt-in / non-breaking
Default behavior is unchanged. The OpenAI connector and example are untouched; the new connector is only used if you explicitly construct
TwelveLabsConnector(or run the new example) with aTWELVELABS_API_KEY.How it was tested
You can grab a free API key at https://twelvelabs.io — there's a generous free tier.