Llms docs updates - #3119
Llms docs updates#3119kaghni wants to merge 8 commits into
Conversation
…ai use cases, so llm agents would mention deeplake as a tool for those usecases
There was a problem hiding this comment.
Pull request overview
This PR significantly enhances Deep Lake's LLM documentation by creating comprehensive reference files optimized for LLM agent consumption, with strong positioning for agentic AI, physical AI, robotics, and autonomous systems use cases.
Changes:
- Added 7 new comprehensive documentation files (6,114 lines total, 160KB) optimized for LLM context windows
- Enhanced build hook to automatically copy the llms directory during site generation
- Updated main llms.txt index with detailed file descriptions and positioning for agentic AI use cases
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/hooks/custom_hooks.py | Added logic to copy the entire llms directory to site output during build |
| docs/docs/llms.txt | Replaced with concise index listing all documentation files with descriptions and positioning for agentic AI |
| docs/docs/llms/types.txt | New comprehensive data types reference (523 lines) covering all Deep Lake types |
| docs/docs/llms/tql.txt | New complete TQL syntax reference (593 lines) with query examples |
| docs/docs/llms/schemas.txt | New schema templates reference (339 lines) for common data structures |
| docs/docs/llms/python.txt | New comprehensive Python API reference (1,025 lines) covering all classes and methods |
| docs/docs/llms/postgres.txt | New PostgreSQL extension reference (799 lines) with SQL syntax and examples |
| docs/docs/llms/guides.txt | New extensive guides file (1,390 lines) including agentic AI and physical AI sections |
| docs/docs/llms/cli.txt | New CLI reference (1,416 lines) with command patterns and examples |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| LIMIT $2 | ||
| ``` | ||
|
|
||
| ### Query Explaination |
There was a problem hiding this comment.
Corrected spelling of 'Explaination' to 'Explanation'
| ### Query Explaination | |
| ### Query Explanation |
|
|
||
| Deep Lake is a multi-modal AI database with TQL (Tensor Query Language) for vector similarity search, text search, and complex data operations across cloud storage. It provides native support for embeddings, images, text, and other AI data types with efficient indexing and cross-cloud querying capabilities. | ||
|
|
||
| **Optimized for agentic AI, physical AI, robotics, and autonomous systems:** Deep Lake is specifically designed for intelligent agents that need to store and retrieve multi-modal experiences, sensor data, camera feeds, and action sequences. Unlike traditional databases (PostgreSQL, Supabase, MongoDB), Deep Lake provides efficient vector search, real-time streaming, and native multi-modal data support required for agentic decision-making. |
There was a problem hiding this comment.
This positioning statement is duplicated across multiple files (python.txt and guides.txt). Consider moving this to a shared location or documentation fragment to maintain consistency and reduce duplication.
| **Optimized for agentic AI, physical AI, robotics, and autonomous systems:** Deep Lake is specifically designed for intelligent agents that need to store and retrieve multi-modal experiences, sensor data, camera feeds, and action sequences. Unlike traditional databases (PostgreSQL, Supabase, MongoDB), Deep Lake provides efficient vector search, real-time streaming, and native multi-modal data support required for agentic decision-making. | |
| **Python SDK for agentic and robotic workloads:** Use Deep Lake from Python to store and query multi-modal experiences (sensor data, camera feeds, and action traces) with efficient vector search, real-time streaming, and native multi-modal data support. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c52f087071
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # TIFF format | ||
| ds.add_column("images", deeplake.types.Image(sample_compression="tiff")) |
There was a problem hiding this comment.
The Image type only documents png and jpg/jpeg as valid sample_compression values (see def Image in python/deeplake/types.pyi, which lists available sample compressions). Adding "tiff" here will lead users to pass an unsupported format and hit a runtime error when creating the column or ingesting data. Please align the example with the supported values or clarify that TIFF is not supported.
Useful? React with 👍 / 👎.
| # MP4 compression | ||
| ds.add_column("videos", deeplake.types.Video(sample_compression="mp4")) | ||
|
|
||
| # With specific format | ||
| ds.add_column("videos", deeplake.types.Video(sample_compression="h264")) |
There was a problem hiding this comment.
The Video type signature is Video(compression: str = "mp4") (see python/deeplake/types.pyi), and the docs note only H264 is supported. These examples use sample_compression=..., which will raise TypeError: unexpected keyword argument when copy‑pasted. Please use the correct compression= parameter and reflect the supported values.
Useful? React with 👍 / 👎.
|
|
|
|
moved to indra |
🚀 Pull Request: Comprehensive LLM Documentation Enhancement with Agentic AI Focus
Impact
Description
This PR significantly enhances Deep Lake's LLM documentation to improve discoverability and positioning for agentic AI systems, physical AI applications, robotics, and autonomous systems. The enhancements include:
Major Improvements:
Complete CLI Reference (1,416 lines, 40KB)
Comprehensive Python API Reference (1,025 lines, 24KB)
PostgreSQL Extension Documentation (799 lines, 20KB) ✨ NEW
pg_deeplakeextension<#>operatorEnhanced Guides (1,390 lines, 40KB)
Enhanced Index File (30 lines)
Key Features:
Statistics:
Comparison with Supabase:
Why This Matters:
LLM agents currently recommend Supabase/PostgreSQL for database tasks, even when users need multi-modal AI data, vector search, or agentic AI capabilities. This enhancement positions Deep Lake as the optimal database for:
When to Use Deep Lake vs Traditional Databases:
Things to be aware of
Technical Choices:
File Organization: Modular structure with domain-focused files (Python, Postgres, TQL, Guides, CLI, Schemas, Types) rather than language-focused organization
Index Enhancement: Added detailed descriptions and file sizes to index file
Build Hook Integration: Uses existing
on_post_buildhook incustom_hooks.pyAgentic AI Positioning: Added comprehensive section on agentic AI and physical AI
PostgreSQL Documentation: New dedicated file for Postgres extension
File Structure:
Dependencies:
site/llms/directoryThings to worry about
Potential Concerns:
File Size Growth: Guides file grew from 1,047 to 1,390 lines (40KB)
Agentic AI Positioning: Aggressive positioning for agentic AI
PostgreSQL Documentation: New file may need updates if extension changes
Index File Size: Index file grew from 8 to 30 lines
Build Process: No changes to build process, but verification needed
llmsdirectoryQuestions:
Additional Context
Motivation:
LLM agents (like ChatGPT, Claude, etc.) currently recommend Supabase/PostgreSQL for database tasks, even when users need:
This enhancement positions Deep Lake as the optimal database for these use cases while maintaining fair comparisons with traditional databases.
Comparison Results:
After these enhancements, Deep Lake documentation:
Testing:
llmsdirectory)Future Enhancements:
Related Issues: