ServeReplica ActorDiedError when TechPredPlugin actor is killed by ray.kill

[touale]

The TechPredPlugin Serve replica exited unexpectedly during task execution, causing an ActorDiedError to be raised to the caller.

Error log:

ActorDiedError
The actor died unexpectedly before finishing this task.

class_name: ServeReplica:default:company_technology_prediction.TechPredPlugin
actor_id: d7ea9e3faddb8916b53dcf5201000000
pid: 6887
name: SERVE_REPLICA::default#company_technology_prediction.TechPredPlugin#u1kq7usq
namespace: serve
ip: 10.233.116.162

The actor is dead because it was killed by `ray.kill`.

[touale]

By default, the Serve controller periodically health-checks each Serve deployment replica and restarts it on failure.

https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html

[touale]

The short-term plan is to optimize health detection, while intensive optimization should focus on resource reuse for plugins, such as allowing several plugins to share the same instance in ray mode.

[touale]

By default, the Serve controller periodically health-checks each Serve deployment replica and restarts it on failure.

https://docs.ray.io/en/latest/serve/production-guide/fault-tolerance.html

health_check_period_s – Duration between health check calls for the replica. Defaults to 10s. The health check is by default a no-op Actor call to the replica, but you can define your own health check using the “check_health” method in your deployment that raises an exception when unhealthy.

health_check_timeout_s – Duration in seconds, that replicas wait for a health check method to return before considering it as failed. Defaults to 30s.