Skip to content

glutton: replace stuck actors on resume/hibernate failure - #1759

Open
Chenyi Wang (chw120) wants to merge 1 commit into
agent-substrate:mainfrom
chw120:fix-glutton-retry-loop-and-actor-reset
Open

Chenyi Wang (chw120) wants to merge 1 commit into
agent-substrate:mainfrom
chw120:fix-glutton-retry-loop-and-actor-reset

Conversation

@chw120

Copy link
Copy Markdown
Collaborator

Summary

Replace broken/stuck gluttonActor instances in place when resume() or hibernate() fails during benchmark iterations.

Problem

When SuspendActor fails (e.g., due to transient GCS ResourceExhausted errors during cold-bucket ramp-up), ate-api-server can leave the actor permanently stuck in ACTOR_STATE_SUSPENDING.

Previously, gluttonUser kept the broken gluttonActor in its u.actors slice for the remainder of the benchmark run:

  • hibernate() ignored SuspendActor / PauseActor errors.
  • Subsequent ResumeActor calls against that slot repeatedly failed with got: ACTOR_STATE_SUSPENDING for the rest of the test, reducing effective concurrency and generating continuous failure noise.

Solution

  • Have hibernate(), pause(), and suspend() return bool indicating whether the operation succeeded.
  • Add (*gluttonUser).replaceActor(ctx, broken) to delete the broken actor (AnyState: true) and replace its slot in u.actors with a freshly created actor (sb-<uuid>).
  • Invoke user.replaceActor(ctx, actor) in iterate() when either actor.resume(ctx) or actor.hibernate(ctx) fails.
  • Add unit test coverage in lifecycle_test.go (TestGluttonIterate_ReplacesActorOnResumeFailure).

It's a good idea to open an issue first for discussion.

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant