From 02cffc0d5e75a645c2c457c946b94b9272bd46bd Mon Sep 17 00:00:00 2001 From: kausmeows Date: Fri, 24 Jul 2026 00:21:21 +0530 Subject: [PATCH] update twelve labs tools for embed_video function --- examples/tools/twelvelabs-tools.mdx | 25 +++++++++++++++- .../examples_sync/description-overrides.json | 2 +- tools/toolkits/others/twelvelabs.mdx | 30 +++++++++++++++++-- 3 files changed, 53 insertions(+), 4 deletions(-) diff --git a/examples/tools/twelvelabs-tools.mdx b/examples/tools/twelvelabs-tools.mdx index c1974a612..7c8ed2921 100644 --- a/examples/tools/twelvelabs-tools.mdx +++ b/examples/tools/twelvelabs-tools.mdx @@ -1,6 +1,6 @@ --- title: "TwelveLabs Tools" -description: "Answer questions about a video URL with TwelveLabs Pegasus analyze_video, and generate Marengo multimodal text embeddings with embed_text." +description: "Answer questions about a video URL with TwelveLabs Pegasus analyze_video, and generate Marengo multimodal embeddings with embed_text and embed_video." source: cookbook/91_tools/twelvelabs_tools.py --- @@ -16,6 +16,11 @@ Demonstrates using TwelveLabs video understanding tools with an agent. `analyze_video` answers questions about a video using the Pegasus model. `embed_text` generates a multimodal (Marengo) embedding that lives in the same latent space as TwelveLabs video/audio/image embeddings. +`embed_video` embeds a whole video into the same Marengo latent space (one vector +per 2-10s segment). It is long-running (async task polling) so it is opt-in +(`enable_embed_video=True`), and it returns a compact summary of the segmentation +(segment count, dimensions and per-segment time offsets) rather than the raw +vectors, which would flood the model context. Set your API key first: `export TWELVELABS_API_KEY=...` Grab a free key at https://twelvelabs.io. @@ -51,6 +56,24 @@ embedding_agent = Agent( embedding_agent.print_response( "Embed the text 'a cat playing piano' and tell me how many dimensions it has." ) + +# Example 3: Embed a whole video with Marengo (one vector per segment). This is +# asynchronous under the hood — the tool waits for the embedding task to finish. +video_embedding_agent = Agent( + tools=[ + TwelveLabsTools( + enable_analyze_video=False, + enable_embed_text=False, + enable_embed_video=True, + ) + ], + markdown=True, +) + +video_embedding_agent.print_response( + "Embed this video and tell me how many segments and dimensions it has: " + "https://interactive-examples.mdn.mozilla.net/media/cc0-videos/flower.mp4" +) ``` ## Run the Example diff --git a/scripts/examples_sync/description-overrides.json b/scripts/examples_sync/description-overrides.json index 5a2953b25..22795fe62 100644 --- a/scripts/examples_sync/description-overrides.json +++ b/scripts/examples_sync/description-overrides.json @@ -922,7 +922,7 @@ "examples/tools/scavio-tools": "Gate ScavioTools providers with enable_* flags to build web-only, commerce-only, and all-provider search agents over Google, YouTube, Amazon, Walmart, Reddit, TikTok, and Instagram.", "examples/tools/searchapi-tools": "Configure SearchApiTools for Google web, News, Images, and YouTube SERP results using per-engine enable flags or all=True.", "examples/tools/sofya-tools": "Toggle SofyaTools between web search, markdown URL extraction, and cited deep-research report generation.", - "examples/tools/twelvelabs-tools": "Answer questions about a video URL with TwelveLabs Pegasus analyze_video, and generate Marengo multimodal text embeddings with embed_text.", + "examples/tools/twelvelabs-tools": "Answer questions about a video URL with TwelveLabs Pegasus analyze_video, and generate Marengo multimodal embeddings with embed_text and embed_video.", "examples/tools/websearch-tools-advanced": "Configure WebSearchTools time limits, regions, and DDGS backends for text and news search.", "examples/tools/x402scan-mcp-tools": "Pay for 100+ paid APIs autonomously with USDC on Base via the x402scan MCP server, with a single agent and a researcher/analyst team sharing one wallet.", "examples/tools/models/azure-openai-tools": "Legacy AzureOpenAITools example for image generation with retired DALL-E deployments.", diff --git a/tools/toolkits/others/twelvelabs.mdx b/tools/toolkits/others/twelvelabs.mdx index 97c3a6d7c..137b6c6a3 100644 --- a/tools/toolkits/others/twelvelabs.mdx +++ b/tools/toolkits/others/twelvelabs.mdx @@ -3,7 +3,11 @@ title: TwelveLabs description: "Analyze videos with the TwelveLabs Pegasus model and generate multimodal text embeddings with Marengo." --- -**TwelveLabsTools** enable an Agent to understand and search video using [TwelveLabs](https://twelvelabs.io). The toolkit exposes two capabilities: `analyze_video`, which answers a natural-language question about a video using the Pegasus video understanding model, and `embed_text`, which generates a multimodal embedding with the Marengo model that lives in the same latent space as TwelveLabs video, audio and image embeddings (useful for searching a video corpus by text). +**TwelveLabsTools** enable an Agent to understand and search video using [TwelveLabs](https://twelvelabs.io). The toolkit exposes three capabilities: + +- `analyze_video`: answers a natural-language question about a video using the Pegasus video understanding model. +- `embed_text`: generates a multimodal embedding with the Marengo model that lives in the same latent space as TwelveLabs video, audio and image embeddings (useful for searching a video corpus by text). +- `embed_video`: embeds a whole video into the same Marengo latent space (one vector per 2–10s segment). Video embedding is asynchronous, so the tool creates an embedding task, polls until it finishes (bounded by `embed_timeout`), and returns a compact summary of the segmentation (segment count, embedding dimensionality, and each segment's time offsets and scope) rather than the raw float vectors. Because it is long-running, `embed_video` is opt-in and disabled by default. ## Prerequisites @@ -53,6 +57,24 @@ embedding_agent = Agent( embedding_agent.print_response( "Embed the text 'a cat playing piano' and tell me how many dimensions it has." ) + +# Example 3: Embed a whole video with Marengo (one vector per segment). This is +# asynchronous under the hood — the tool waits for the embedding task to finish. +video_embedding_agent = Agent( + tools=[ + TwelveLabsTools( + enable_analyze_video=False, + enable_embed_text=False, + enable_embed_video=True, + ) + ], + markdown=True, +) + +video_embedding_agent.print_response( + "Embed this video and tell me how many segments and dimensions it has: " + "https://interactive-examples.mdn.mozilla.net/media/cc0-videos/flower.mp4" +) ``` ## Toolkit Params @@ -61,10 +83,13 @@ embedding_agent.print_response( | ---------------------- | --------------- | -------------- | ----------------------------------------------------------------------------------------------- | | `api_key` | `Optional[str]` | `None` | The TwelveLabs API key. Read from the `TWELVELABS_API_KEY` environment variable if not provided. | | `analyze_model` | `str` | `pegasus1.5` | The Pegasus model used for `analyze_video`. | -| `embed_model` | `str` | `marengo3.0` | The Marengo model used for `embed_text`. | +| `embed_model` | `str` | `marengo3.0` | The Marengo model used for `embed_text` and `embed_video`. | | `max_tokens` | `int` | `2048` | Maximum number of tokens for `analyze_video` responses. | +| `embed_poll_interval` | `float` | `5.0` | Seconds to wait between status checks while an `embed_video` task is processing. | +| `embed_timeout` | `float` | `300.0` | Maximum seconds to wait for an `embed_video` task to finish before giving up. | | `enable_analyze_video` | `bool` | `True` | Enable the `analyze_video` functionality. | | `enable_embed_text` | `bool` | `True` | Enable the `embed_text` functionality. | +| `enable_embed_video` | `bool` | `False` | Enable the `embed_video` functionality. Opt-in because video embedding is long-running. | | `all` | `bool` | `False` | Enable all functionality. | ## Toolkit Functions @@ -73,6 +98,7 @@ embedding_agent.print_response( | --------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `analyze_video` | Analyze a video and answer a natural-language question about it using the Pegasus model. | | `embed_text` | Generate a multimodal (Marengo) embedding for a piece of text, which can be used to search a video corpus by text. | +| `embed_video` | Embed a whole video into the Marengo latent space (one vector per 2–10s segment) and return a summary of the segmentation. | ## Developer Resources