Improve ComfyUI Integration - Local & Cloud Support, Workflow Preview, Community Browser

[yihangwu539-png]

Summary

This PR significantly improves the ComfyUI integration in Clapper, addressing issue #5. It fixes existing bugs, completes incomplete implementations, and adds new features for workflow preview and community workflow browsing.

Changes Made

1. 🔧 Fixed Local ComfyUI Provider (packages/app/src/app/api/resolve/providers/comfyui/index.ts)

• Fixed the category-to-workflow mapping to support all segment types (IMAGE, VIDEO, SOUND, VOICE, DIALOGUE, MUSIC)
• Added getWorkflowForSegment() helper function to properly look up the correct workflow from settings
• Added getWorkflowCategoryForSegment() helper to map segment categories to workflow categories
• Improved error handling with clearer error messages
• Graceful warning instead of throwing when prompt input is not explicitly configured
• Simplified output asset extraction logic
• Removed the broken catch-all error that prevented IMAGE/VIDEO generation from working properly

2. 🔧 Fixed Comfy.icu Cloud Provider (packages/app/src/app/api/resolve/providers/comfy-comfyicu/index.ts)

• Complete rewrite of the workflow execution: Previously it just started a run and immediately threw an error saying "not supported"
• Added pollWorkflowRun() - a robust polling mechanism that checks workflow status every 2 seconds (configurable)
• Added support for all workflow categories: IMAGE, VIDEO, SOUND, VOICE, DIALOGUE, MUSIC
• Proper error handling for API failures, timeouts, and failed/cancelled runs
• Added proper output asset URL extraction from completed runs
• Added sleep() utility for polling intervals
• Updated types to support run_id field and additional payload fields

3. 🆕 Community Workflow API Routes (packages/app/src/app/api/workflows/)

New API endpoints for browsing community workflows:

GET /api/workflows?q=...&source=openart&page=1

• Search/browse workflows from OpenArt.ai and CivitAI
• Normalizes responses from different sources into a common format
• Gracefully handles API unavailability with fallback empty results

GET /api/workflows/[source]/[id]

• Fetch a specific workflow by ID from a given source
• Returns full workflow details including the actual workflow JSON data (when available)

4. 🆕 Workflow Graph Preview Component (packages/app/src/components/comfy/ComfyWorkflowGraphPreview.tsx)

A React component that renders a ComfyUI workflow graph as a visual SVG node graph:

• Parses the ComfyUI API JSON format (nodes, edges, class types)
• Color-codes nodes by type (blue=load/input, purple=sampler, green=encode/decode, amber=output, pink=conditioning, cyan=transformation)
• Uses topological sort (Kahn's algorithm) for automatic layout
• Shows node labels and connection lines
• Displays node count and connection count
• Configurable dimensions and max nodes
• Handles malformed workflow data gracefully

5. 🆕 Community Workflow Browser Component (packages/app/src/components/comfy/CommunityWorkflowBrowser.tsx)

A UI component for browsing and importing workflows from community platforms:

• Search bar with query input
• Source selector (OpenArt.ai / CivitAI)
• Workflow list with thumbnails, author, description, tags, node count, likes
• Visual selection state for the chosen workflow
• "Apply Workflow" button to trigger the import
• Loading state, error state, and empty state handling

6. 🆕 Component Exports (packages/app/src/components/comfy/index.ts)

Barrel exports for the new ComfyUI components.

Implementation Details

Local ComfyUI

The local provider connects to any ComfyUI instance (default: http://localhost:8188) using the @saintno/comfyui-sdk. It supports HTTP Basic Auth for secured instances. Workflows are configured in settings and the provider automatically maps Clapper inputs (prompt, negative prompt, width, height, seed, image) to the workflow node inputs.

Comfy.icu Cloud

The cloud provider uses the Comfy.icu API with a polling workflow:

1. POST /api/v1/workflows/{id}/runs to start a run
2. GET /api/v1/workflows/{id}/runs/{runId} to poll for status
3. Extract output URLs from completed runs
4. Configurable polling interval (2s) and timeout (4 min)

Community Workflow Sources

• OpenArt.ai: Searches via their workflow search endpoint. When a workflow is found, it can be fetched to get the full ComfyUI workflow JSON.
• CivitAI: Uses the public CivitAI API to search models. ComfyUI workflows are fetched when available.

Testing

• ✅ Local ComfyUI provider: category mapping, error handling, prompt injection
• ✅ Comfy.icu provider: polling mechanism, status tracking, timeout handling
• ✅ Workflow graph preview: parsing, layout, rendering all node types
• ✅ Community workflow browser: search, select, apply flow
• ✅ API routes: parameter handling, error responses, source switching

Future Improvements

• Support for ComfyDeploy cloud provider (separate from Comfy.icu)
• Better graph layout algorithms (Dagre-like layered rendering)
• Direct drag-and-drop workflow import from the browser
• Workflow editor integration for modifying community workflows before applying
• Support for loading workflows from local filesystem