feat(proxy): reject at capacity instead of queueing unboundedly

[nilsmechtel]

What

When an app's ProxyDeployment semaphore is full, the per-method proxy function now raises a RuntimeError telling the caller the app is at capacity, instead of silently awaiting a free slot.

Application '<id>' has reached its maximum of N concurrent requests and is currently very busy. Please try again in a moment.

The cap is the existing per-app max_ongoing_requests knob (deploy-time deploy_app(max_ongoing_requests=…), default 10; decorator default @bioengine.app(max_ongoing_requests=…)). No new parameter.

Why

Every user call (WebSocket + WebRTC) funnels through deployment_function, which held async with self.service_semaphore. When full, request N+1 awaited a slot — an unbounded queue of coroutines, each already holding its decoded payload (e.g. an inference image) in the proxy actor's memory. Under load this grows proxy memory without bound and gives the caller no signal. Failing fast returns capacity pressure to the caller and frees the payload immediately.

Behaviour changes from "queue when full" to "reject when full" for every app. It does not touch the inner Entry/Runtime deployments' own Ray Serve max_ongoing_requests (the autoscaling-signal queue) — only the proxy's Hypha-facing semaphore.

Safety

.locked() is race-free here: because we reject rather than await, the semaphore never accumulates waiters, so .locked() reduces to "no free slot", and there is no await between the check and the acquire.

Validation

Deployment-side change → validated on a dev image on a live cluster before marking ready (deploy an app with max_ongoing_requests=1, fire concurrent calls, confirm the excess returns the busy RuntimeError while in-flight calls still succeed).

🤖 Generated with Claude Code