Allow actors to self-suspend

[juli4n]

Substrate's value comes from suspending idle actors and multiplexing many of them onto few workers. But today nothing decides when an actor is idle: suspend is only triggered by an external SuspendActor/PauseActor API call, presumably a higher order system.

We discussed an automatic idle-detection, but doesn't exist yet and I'm not sure if it can realistically work for session-based protocols such as websockets or grpc (with stream).

For some use cases, actors reliably know when they are idle, "I finished my turn / my streams are dormant / checkpoint me."

We should consider a cooperative model where actors yield by writing to a well-known file (e.g. /var/ate/state/suspend).

[EItanya]

This feels like a dupe of #257?

[juli4n]

Yes, both seem to propose the same feature, they differ on the mechanism to accomplish it, but better to merge and have a single discussion.

[ronlv10]

+1 for this feature.

I'd like to add that calling SuspendActor doesn't reliably work right now. When the actor calls the ate-api, the actor gets suspended, and upon resuming, an HTTP error occurs because the (previous) connection dropped (I talked about this issue here). This leaves no good way to verify if the suspension actually succeeded. I think writing to a file instead would make the process more atomic and robust.

[sfunkenhauser]

Actor idleness probe

I propose adding an idleness probe, instead of a self-suspend action.

Why not self-suspend

The obvious way to let an actor give its worker slot back is to let it ask: the actor tells ateom it has nothing left to do, and ateom proxies the suspend. The practical problem is that there is no actor→host channel today, and adding one is not cheap: HTTP means configuring the actor's egress and pinning a static endpoint IP:port into the sandbox, vsock exists only on some runtimes and would need a UDS on gVisor, and either way the actor carries client code and a protocol it did not previously need. That makes the actor non-portable — the workload has to know it is running on Substrate (and potentially which sandbox class).

The deeper problem is that self-suspend asserts a transition rather than a fact, and a transition is a lifecycle the actor then has to own and manage: stop serving, request the suspend, wait to be woken, resume serving — along with whatever edge cases fall out of the request being asynchronous, since the actor goes on running for an indeterminate time after asking.

Rather than the actor asserting "suspend me," the platform asks "are you idle?" and the actor answers. The answer is a fact about the actor, which the actor is the only party qualified to state; the decision remains the platform's. The actor needs no client, no protocol, and no knowledge that Substrate exists beyond serving one more HTTP endpoint on the interior network it already serves readyz on.

Design

API

One optional field on Container, alongside readyz:

// idle_probe is an optional HTTP probe the platform polls to ask whether the
// container is idle. 200 means idle; any other status or a failure to
// connect means busy. path defaults to "/idlez".
//
// +k8s:optional
HTTPGetAction idle_probe = N;

There is deliberately no threshold, no period, and no minimum-uptime configuration. The response is a bare boolean and the actor integrates whatever duration or heuristic it wants behind it. This keeps the platform from second-guessing an answer it has strictly less information to produce, and lets each workload define "idle" for itself.

Data path

1. ateom polls. While the actor is RUNNING, its ateom GETs /idlez over the existing interior veth, using the same client shape as the readyz probe. Nothing polls a SUSPENDED actor — it has no sandbox to dial, and a suspended actor is idle by definition.
2. ateom proposes. On a 200, ateom requests a suspend through the broker chain it already uses: ateom → atelet → ateapi.
3. ateapi disposes. The suspend is a proposal, and the control plane has the final veto — the same one it already applies to any suspend. Keeping the decision there means other signals can be folded in: an in-progress request that arrived after the idle probe fired, for example, could veto the suspend. Additional policy can be added in the future.

The split is that the data plane detects and proposes; the control plane decides.

Integrating

Serve one endpoint. Return 200 when you have nothing in flight, 503 otherwise:

func (a *app) idlez(w http.ResponseWriter, r *http.Request) {
    if a.busy() {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

Use readyz to avoid suspend loops. An actor that reports idle immediately after waking can be suspended again before it ever receives the request that woke it. readyz is the signal that prevents this: it is polled on every activation — first run and every restore:

readyz 200 ──▶ Activate ──▶ ateom → atelet → ateapi ──▶ RUNNING ──▶ atenet unparks ──▶ first request
           └──────────────── actor is up but not yet reachable ────────────────┘

So arming "busy" from the readyz handler guarantees the actor is marked busy before any /idlez poll and before the first post-resume request arrives:

func (a *app) readyz(w http.ResponseWriter, r *http.Request) {
    a.armBusy() // idempotent; readyz may be polled many times per activation
    w.WriteHeader(http.StatusOK)
}

The actor then decides when to disarm — after the first request completes, after a quiet period, or on whatever condition matches its workload. The platform stays out of it.

Known gap: suspend cancels in-flight requests

Suspend does not drain today, it cancels. atunnel's Deactivate rejects new requests, cancels the requests already in flight for the active actor, and waits for their handlers to exit.

That leaves a race here. The probe can report idle and a request can arrive before the freeze lands, and that request is canceled. Narrowing the window does not close it: idleness is a fact about the past the moment it is reported, so some interval always exists between the answer and the freeze.

This is not specific to the probe. Anytime suspend is triggered this can be an issue. The probe adds a new trigger, not a new failure mode. The fix is lifecycle hooks: a pre-checkpoint hook that lets the actor finish or shed what it is holding before the freeze. That is tracked by #450.

[thockin]

Serve one endpoint. Return 200 when you have nothing in flight, 503 otherwise:

What does it mean to get any other code?

ateom polls. While the actor is RUNNING, its ateom GETs /idlez over the existing interior veth, using the same client shape as the readyz probe.

Polls how often? Do we need any guarantees on this? If I have a machine with "task runner" actors on it, I don't have visibility into when a task starts/finishes, so this polling loop becomes critical in driving overall efficiency / density. If I have 1000 such actors, do I poll once per second? That's 1000QPS of traffic being sent, and 1 second idle detection isn't really that impressive...

Did we consider something like a long-poll model? Ateom does a GET on /idlez and the actor does not respond until it actually is idle (or times out). Ateom needs to keep the socket(s) open but it can select() on input rather than poll.

The actor then decides when to disarm — after the first request completes, after a quiet period, or on whatever condition matches its workload. The platform stays out of it.

Are we guaranteeing that we will never look at idlez while readyz is false? Are we guaranteeing that we will not strobe readyz at any other time besides right after wakeup? Do we need a way to signal "I just woke you up" instead?

I'm concerned that we will hit readyz in other cases and trigger actors to think they are busy, and then they don't properly idle. I'd like to see the "is is idle" state machine which includes both probes.

If we go this way, we should provide a Go library which implements and demonstrates the correct behavior.

[EItanya]

Going to start by saying I don't have a great solution to the problem I'm going to lay out, but thinking it through out loud.

I'm not sure idlez would fix the problem unless we're willing to pull the polling time down to near 0. Let's take the use case outlined during the offsite of suspending while waiting for LLM or any long-running HTTP/IO call to return. O(250ms), (a potential polling time) could already be a sizable chunk of that call.

In this case, whatever broker is managing the handoff of said LLM/HTTP calls for the actor could suspend them, but just wanted to think through the timing guarantees we're trying to have for self-suspending