Add bounded concurrent run queueing for per-run stack startups

[emmepra]

Problem / Opportunity

Submitting multiple per-run environment stacks concurrently is not robust enough on the current platform path.

In MASE-internal, launching several runs at once on the MIND study stack caused one run to start cleanly while the next failed during stack creation with terminal reason:

• Per-run services failed health check: environment-backend service failed health check

The controller accepted the runs, but the orchestrator attempted to start too many sibling run stacks immediately. The safe workaround was to switch to strictly sequential launches. That is acceptable as an experiment workaround, but it is a platform limitation: the platform should support bounded concurrent runs predictably.

Desired Outcome

The controller/orchestrator path should support multiple concurrent runs without fragile startup races.

Specifically:

• the platform should accept many run submissions
• launch only up to a configured concurrency cap
• queue excess runs instead of trying to start all run stacks immediately
• avoid failing runs solely because sibling per-run stacks are also starting

Proposed Direction

• Add a controller/orchestrator concurrency cap for per-run stack startups.
• Treat accepted-but-not-yet-started runs as queued/pending rather than attempting immediate startup.
• Add explicit startup backpressure and/or jitter between run-stack launches.
• Improve failure visibility so health-check failures surface with enough context to debug resource contention.
• Consider separate caps for run submission, stack startup, and active running stacks.

Scope

• In scope: per-run stack orchestration, startup queueing, health-check robustness, controller/orchestrator coordination.
• Out of scope: study-specific launcher defaults, MIND-only analysis code, provider quota handling.

Acceptance Criteria

• When multiple runs are submitted quickly, only up to the configured startup concurrency cap begin stack creation at once.
• Additional accepted runs remain queued/pending instead of failing due to concurrent startup pressure.
• A healthy run no longer fails solely because a sibling run is starting at the same time.
• Queue/startup status is visible through controller APIs or logs.
• Health-check failure logs clearly identify which service failed and during which orchestration phase.

Context

• Source: ongoing MIND social-proof study on MASE-internal.
• Related: merged PR #6 added bounded HTTP response snapshots for telemetry and made fresh reruns useful for bandwagon-style analysis.
• Observed behavior:
  • concurrent --launch-all path caused environment-backend health-check failure on the second sibling run
  • switching to sequential launches avoided the failure and let the same workload progress cleanly