feat: comprehensive ComfyUI integration for all media types#137
Open
pannous wants to merge 3 commits into
Open
feat: comprehensive ComfyUI integration for all media types#137pannous wants to merge 3 commits into
pannous wants to merge 3 commits into
Conversation
Fixes jbilcke-hf#5 - Extended ComfyUI local provider to support voice, sound, and music generation (previously only image and video were supported) - Fixed ComfyICU provider with proper submit/poll flow and multi-category support (was a broken stub that always threw errors) - Fixed ComfyDeploy provider with submit/poll flow, prompt injection, and multi-category support (was a broken stub) - Added ClapWorkflow types and default workflows for voice/sound/music - Added setters for comfyClapWorkflowForVoice/Sound/Music in settings store - Upgraded voice/sound/music settings UI from raw text areas to use FormComfyUIWorkflowSettings component (consistent with image/video) - Added ComfyUI connection status indicator in provider settings - Added ComfyDeploy API key field to provider settings UI - Extended convertComfyUiWorkflowApiToClapWorkflow for all audio categories - Extended getMainInputIdsByClapWorkflowCategory with audio-specific inputs
There was a problem hiding this comment.
Pull request overview
This PR expands the ComfyUI ecosystem integrations so Clapper can run ComfyUI workflows for all segment media types (image, video, voice, sound, music), and replaces the previously non-functional Comfy.icu and ComfyDeploy provider stubs with real submit/poll execution flows.
Changes:
- Adds default ComfyUI
ClapWorkflowsettings + Zustand setters for voice/sound/music, and updates the settings UI to use the same workflow editor component as image/video. - Extends the local ComfyUI resolver to route workflows by segment category and handle audio outputs.
- Implements Comfy.icu and ComfyDeploy workflow execution via run submission + polling, with prompt injection and output URL extraction.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/clapper-services/src/settings.ts | Extends settings types with ComfyUI clap-workflow slots for voice/sound/music. |
| packages/app/src/services/settings/workflows/voice.ts | Adds default ComfyUI clap workflow definition for voice. |
| packages/app/src/services/settings/workflows/sound.ts | Adds default ComfyUI clap workflow definition for sound. |
| packages/app/src/services/settings/workflows/music.ts | Adds default ComfyUI clap workflow definition for music. |
| packages/app/src/services/settings/getDefaultSettingsState.ts | Wires new default clap workflows into the default settings state. |
| packages/app/src/services/settings/useSettings.ts | Adds setters and persistence/merge wiring for new ComfyUI clap workflows. |
| packages/app/src/components/settings/voice.tsx | Switches voice settings from raw JSON textarea to FormComfyUIWorkflowSettings. |
| packages/app/src/components/settings/sound.tsx | Switches sound settings from raw JSON textarea to FormComfyUIWorkflowSettings. |
| packages/app/src/components/settings/music.tsx | Switches music settings from raw JSON textarea to FormComfyUIWorkflowSettings. |
| packages/app/src/components/settings/provider.tsx | Adds ComfyUI connection status indicator and ComfyDeploy API key field. |
| packages/app/src/components/comfyui/ComfyUIConnectionStatus.tsx | New UI component that polls /system_stats to show local ComfyUI connectivity. |
| packages/app/src/app/api/resolve/providers/comfyui/index.ts | Extends local ComfyUI resolver routing + inputs/outputs to support audio categories. |
| packages/app/src/app/api/resolve/providers/comfyui/getMainInputIdsByClapWorkflowCategory.ts | Adds voice/sound/music main input IDs (prompt/seed/output). |
| packages/app/src/app/api/resolve/providers/comfyui/convertComfyUiWorkflowApiToClapWorkflow.ts | Extends ComfyUI-workflow-to-clap conversion metadata for voice/sound/music. |
| packages/app/src/app/api/resolve/providers/comfy-comfyicu/index.ts | Implements Comfy.icu workflow run submission + polling and output URL extraction. |
| packages/app/src/app/api/resolve/providers/comfy-comfydeploy/index.ts | Implements ComfyDeploy workflow run submission + polling and output URL extraction. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The inputValues union type (string | number | boolean | ... | ClapInputValueObject) requires proper narrowing before accessing .id. Use object type guard instead of optional chaining to satisfy TypeScript and prevent potential runtime errors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #5
This PR extends ComfyUI support to all media types (image, video, voice, sound, music) and fixes the broken ComfyICU and ComfyDeploy provider stubs.
Changes
ComfyUI Local Provider (
comfyui/index.ts)audio,files, andimagesoutput formatsComfyICU Provider (
comfy-comfyicu/index.ts)ComfyDeploy Provider (
comfy-comfydeploy/index.ts)Settings & Types
ClapWorkflowtypes for voice/sound/music (comfyClapWorkflowForVoice/Sound/Music)convertComfyUiWorkflowApiToClapWorkflowwith voice/sound/music metadatagetMainInputIdsByClapWorkflowCategorywith audio-specific input IDsUI Improvements
FormComfyUIWorkflowSettingscomponent (consistent with image/video)/system_stats)Test plan
tests.spec.ts)clap,timeline,clapper-services)