Skip to content

Graceful shutdown is broken: no signal handling, dispatcher can deadlock on cancel, web.Serve shuts down with a canceled context #270

Description

@whywaita

Problem

Three related lifecycle issues mean myshoes cannot shut down cleanly.

1. No signal handling at all

Run (cmd/server/cmd.go:81) uses errgroup.WithContext(context.Background()) and nothing installs a signal handler (signal.NotifyContext). SIGTERM/SIGINT kill the process immediately: in-flight AddInstance calls to providers are aborted mid-way, which is a recipe for orphaned VMs.

2. Dispatcher send ignores context cancellation (deadlock)

dispatcher (pkg/starter/starter.go:126-129) sends jobs on an unbuffered channel without a ctx.Done() case:

for _, j := range jobs {
    ch <- j
}

The only receiver is run() (pkg/starter/starter.go:134-188), which returns on ctx.Done(). If the context is canceled while the dispatcher is blocked in ch <- j, the receiver is gone, the send blocks forever, and eg.Wait() (pkg/starter/starter.go:113) never returns.

3. web.Serve shuts down with the already-canceled context and leaks a goroutine

pkg/web/http.go:99-104:

select {
case <-ctx.Done():
    return s.Shutdown(ctx)      // ctx is already canceled → no graceful drain
case err := <-errCh:
    ...
}
  • s.Shutdown(ctx) with a canceled context aborts immediately instead of draining in-flight requests; it needs a fresh timeout context.
  • After Shutdown, ListenAndServe returns http.ErrServerClosed, and the serving goroutine blocks forever on errCh <- ... (unbuffered, nobody reading) — a goroutine leak, and ErrServerClosed shouldn't be treated as an error anyway.

Suggested fix

  • Wrap the root context with signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt) in cmd/server.
  • Make the dispatcher send select { case ch <- j: case <-ctx.Done(): return ctx.Err() }.
  • In web.Serve, use a buffered errCh, ignore http.ErrServerClosed, and call Shutdown with a fresh context.WithTimeout.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions