Say can#7
Open
libgoncalv wants to merge 5 commits into
Open
Conversation
Introduce saycan experiment package for language-conditioned robotic manipulation. Adds README and modules to integrate CLIPort (TransporterNet + CLIP), ViLD object detection, an Ollama-backed LLM interface, PyBullet UR5e environment, dataset utilities, and helper functions. Includes asset downloader and configuration (config.py), a CLIPort wrapper with checkpoint loading/migration logic (cliport.py), a SHARPIE-friendly environment wrapper that combines LLM planning, affordance scoring and CLIPort execution (environment.py), dataset collection code (datasets.py), and utility helpers (helpers.py).
Make environment and model behavior more robust and production-friendly: - README: Update example defaults (wait_for_inputs -> False, inputs_type -> 'other'). - cliport.py: Remove interactive/demo run_cliport helper to avoid shipping debug UI code. - config.py: Call download_assets() on import rather than only when executed as __main__. - environment.py: - Add cv2 import and convert cached frames / camera images to RGB in render(). - Replace ad-hoc 'task:'/plan handling with run_task(), which sets a task and automatically plans+executes steps up to a limit, returning a summary (steps, total_reward, termination_reason) and caching video frames for rendering. - Adjust step handling to return run_task results for 'task:' actions. - vild.py: - Replace eager CLIP/TF loading with lazy getters (get_clip_model, get_tf_session) and a cleanup_models() helper to free resources. - Add VILD_LAZY_LOAD env var to opt into lazy loading; otherwise behavior remains backward-compatible. - Update build_text_embedding to obtain the CLIP model lazily, move embeddings to CPU promptly, and clear GPU cache after use to reduce memory spike. Has been tested on an Nvidia A16 but ViLD still crashes when running multiple times
florisdenhengst
approved these changes
Feb 25, 2026
Collaborator
florisdenhengst
left a comment
florisdenhengst
left a comment
There was a problem hiding this comment.
Please consider:
- adding a Runner object insert to the README
- factoring out the SayCan-specific logic into a separate file for environment.py
| Assets (robot URDFs, ViLD model, CLIPort checkpoint) are downloaded automatically on first run. | ||
|
|
||
| ## Setting on the Webserver | ||
|
|
Collaborator
There was a problem hiding this comment.
for completeness it would be nice to have a runner here too
Collaborator
There was a problem hiding this comment.
this file mixes (a lot of) saycan-specific logic with (some) SHARPIE logic.
Perhaps it would be more illustrative to separate these in order to clarify what SHARPIE needs vs what SayCan needs.
See this PR for an (untested) suggestion/direction SayCan...saycan-split-env
Proposal to factor out base env and SHARPIE env wrapper
Add directory to path for imports in environment.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Can be merged after hybrid-intelligence/SHARPIE#76