Skip to content

fix(docker): deliver SIGTERM to the Ballista processes in the images - #2590

Merged
andygrove merged 13 commits into
apache:mainfrom
villebro:fix/docker-entrypoints-sigterm
Oct 15, 2026
Merged

andygrove merged 13 commits into
apache:mainfrom
villebro:fix/docker-entrypoints-sigterm

Conversation

@villebro

@villebro villebro commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

With this change, containers built from the published images stop the way their Ballista processes expect. Today each entrypoint runs its binary as a child of bash, which stays PID 1 and drops the SIGTERM from docker stop or the kubelet. Executors never drain, and every container waits out its grace period to be SIGKILLed.

Closes #2589.

What changes are included in this PR?

  • Entrypoints. The executor, scheduler and CLI entrypoints exec their binary, so it runs as PID 1 and receives SIGTERM, as the result service's entrypoint already does.
  • SIGTERM handling in each binary. As PID 1, a process only stops on SIGTERM if it handles the signal itself.
    • The executor already handles it, and drains.
    • The scheduler now exits on SIGTERM or Ctrl-C. It doesn't drain its connections yet (Scheduler graceful shutdown #565).
    • The CLI now exits on SIGTERM, leaving the TUI's raw mode and alternate screen first if it's open. Its REPL writes each history entry as it's added, so the history survives any stop. Ctrl-C stays with the REPL.
    • The scheduler shares the result service's handler, which moves to ballista_core::utils::shutdown_signal. ballista_result_service::shutdown_signal re-exports it.
  • Standalone. The entrypoint stops the executor first, so it drains while the scheduler is still up, then stops the scheduler, and passes on the executor's exit status. It waits for the scheduler's port with bash's /dev/tcp, because the image doesn't install nc.
  • Compose. The benchmarks container gets init: true for its sleep infinity.
  • Docs.
    • The Kubernetes guide notes that two settings cap an executor's drain at 30 seconds by default: the pods' terminationGracePeriodSeconds and the scheduler's --executor-termination-grace-period. Raise both if tasks can run longer.
    • The 56.0.0 upgrade guide describes the new stop behaviour.

Related: #565 (the scheduler now exits on SIGTERM; draining is still to come), #2357 (with #2448, executor pods also drain their Flight streams), #2525.

Are there any user-facing changes?

Yes, all in docs/source/upgrading/56.0.0.md:

  • Executors drain their running tasks on SIGTERM.
  • The scheduler, standalone and CLI containers stop promptly on SIGTERM.
  • Outside the images too: the binaries handle SIGTERM themselves, so deployments that override the entrypoint, for example with a Kubernetes command:, behave the same.

How was this tested?

  • shellcheck passes on every changed script.
  • The standalone entrypoint's stop logic was exercised with stand-in processes:
    • on SIGTERM, the executor drains before the scheduler stops
    • the executor's exit status is passed on
    • a SIGTERM during startup stops the scheduler cleanly.
  • On SIGTERM, the CLI exits with status 143 through its own handler, not the signal's default action, and the lines entered before it are in .history. Normal \q exits across sessions keep each entry once.
  • The scheduler's SIGTERM handling wasn't exercised locally, because the sandbox it was developed in blocks port binds. It uses the result service's handler.
  • The reproduction in Docker entrypoints swallow SIGTERM, so graceful shutdown never runs in the published images #2589 uses the same ubuntu:26.04 base and entrypoint shape. Without exec, docker stop never reaches the process, which is SIGKILLed (exit 137). With exec, a process that handles SIGTERM gets it and stops straight away.

Comment thread ballista-cli/src/main.rs Outdated
The entrypoints ran each binary as a child of bash, which stayed PID 1.
Bash as PID 1 has no SIGTERM handler, so the signal from `docker stop`
or the kubelet never reached the executor or scheduler. Executors never
drained, and every container waited out its grace period to be
SIGKILLed.

The entrypoints now `exec` their binary. The images run under tini,
which passes signals on: the executor's SIGTERM handler starts its
graceful shutdown, and the scheduler, which has no handler, exits on
SIGTERM instead of waiting for SIGKILL. The standalone entrypoint stops
the executor first and the scheduler after it, and waits for the
scheduler's port with bash's /dev/tcp, since the image has no `nc`. The
compose file's benchmarks container gets an init process for its
`sleep`.

Part of apache#2589.
The Docker guide shows tini in the `docker ps` output and says how to
give draining executors time to stop. The Kubernetes guide explains what
an executor does on SIGTERM, and that `terminationGracePeriodSeconds`
and the scheduler's `--executor-termination-grace-period` should cover
the longest-running tasks.
After building the images, the Docker workflow runs a scheduler and an
executor, and the standalone image, and stops each with `docker stop`.
It fails if a container is SIGKILLed or takes close to the stop timeout,
and checks that the executor shut down gracefully.

Closes apache#2589.
The `exec` and tini changes can be checked by reading them, so a CI job
that starts containers and depends on log lines and timing costs more
than it adds. The Docker guide's stop section described expected
behaviour. The Kubernetes guide keeps the part that isn't obvious: the
pod's termination grace period and the scheduler's executor grace period
both cap the drain at 30 seconds by default.
Each binary is to handle SIGTERM itself, so it stops on the signal as PID 1,
whether the container runs the image's entrypoint or another command. The
entrypoints' exec is then enough, and the images don't need tini.
The scheduler installed no signal handlers. Outside a container, SIGTERM's
default action stopped it. As PID 1 in a container that default doesn't
apply, so `docker stop` and the kubelet waited out the grace period to
SIGKILL it. The binary now exits on SIGTERM or Ctrl-C itself. It doesn't
drain its connections yet (apache#565).
As PID 1 in a container, SIGTERM's default action doesn't apply, so
`docker stop` waited out its timeout to SIGKILL the CLI. The CLI now exits
on SIGTERM itself, with the status the default action would give. Ctrl-C is
left to the REPL.
The scheduler had a copy of the result service's `shutdown_signal`.
It now lives in `ballista_core::utils`, which both use.
`ballista_result_service::shutdown_signal` re-exports it.
@villebro
villebro force-pushed the fix/docker-entrypoints-sigterm branch from 0d3dd03 to af53a26 Compare October 14, 2026 17:59
…SIGTERM

The SIGTERM handler exits straight away, which skipped the REPL's history
save at the end of its loop and the TUI's terminal restore. The REPL now
writes each history entry as it's added, so the history survives any stop.
The handler leaves the TUI's raw mode and alternate screen before exiting.
@villebro
villebro marked this pull request as ready for review October 14, 2026 22:08
@andygrove

Copy link
Copy Markdown
Member

Thanks @villebro, and thank you @avantgardnerio for reviewing

@andygrove
andygrove merged commit 4f9655f into apache:main Oct 15, 2026
27 checks passed
@villebro
villebro deleted the fix/docker-entrypoints-sigterm branch October 16, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development-process documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docker entrypoints swallow SIGTERM, so graceful shutdown never runs in the published images

4 participants