OPL: Multi-Host Job Distribution (Load Balancing)

[DigiJoe79]

Problem

Currently, the TTS/Quality workers use a single "default" engine and process jobs sequentially. Users with multiple GPU hosts (e.g., local RTX 3090 + remote A100 server) cannot utilize them in parallel. All jobs queue behind one engine while other capable engines sit idle.

Proposed Solution

Enable parallel job processing across multiple hosts running the same engine type.

Core Changes:

1. Engine Pool per Type
   - Track all enabled engines of each type, not just the default
   - Monitor engine status: idle, busy, unavailable
2. Job Dispatcher
   - Replace sequential processing with async job dispatch
   - On new job: find first idle engine of required type
   - If all busy: queue job, dispatch when engine becomes idle
3. Concurrent Worker Threads
   - Configurable parallelism (e.g., max_concurrent_jobs: 4)
   - One thread per active job, not per engine
4. Sticky Sessions (Optional)
   - Keep same speaker/model loaded on engine for related segments
   - Reduces model reload overhead for chapter batches

API Changes

• POST /tts/generate - No change (job queued as before)
• GET /engines/{type}/pool - New: list engines with current status
• Settings: tts.maxConcurrentJobs, tts.preferLocalFirst

UI Changes

• Jobs view: show which host is processing each job
• Engine card: show current load (idle/busy/queue depth)

Non-Goals (v1)

• Dynamic scaling (auto-start engines on demand)
• Cost-based routing (prefer cheaper hosts)

Open Questions

• Failure handling: retry on same host or failover to another?
• Priority: should local engines be preferred over remote (latency)?
• Model affinity: how aggressively keep models loaded?

Testing Requirements

• Minimum 2 GPU hosts with same engine installed
• Simulate slow engine (debug-tts with artificial delay)
• Verify job distribution under load