Repository navigation
fix(docker): deliver SIGTERM to the Ballista processes in the images - #2590
Merged
Merged
Conversation
avantgardnerio
approved these changes
Oct 14, 2026
The entrypoints ran each binary as a child of bash, which stayed PID 1. Bash as PID 1 has no SIGTERM handler, so the signal from `docker stop` or the kubelet never reached the executor or scheduler. Executors never drained, and every container waited out its grace period to be SIGKILLed. The entrypoints now `exec` their binary. The images run under tini, which passes signals on: the executor's SIGTERM handler starts its graceful shutdown, and the scheduler, which has no handler, exits on SIGTERM instead of waiting for SIGKILL. The standalone entrypoint stops the executor first and the scheduler after it, and waits for the scheduler's port with bash's /dev/tcp, since the image has no `nc`. The compose file's benchmarks container gets an init process for its `sleep`. Part of apache#2589.
The Docker guide shows tini in the `docker ps` output and says how to give draining executors time to stop. The Kubernetes guide explains what an executor does on SIGTERM, and that `terminationGracePeriodSeconds` and the scheduler's `--executor-termination-grace-period` should cover the longest-running tasks.
After building the images, the Docker workflow runs a scheduler and an executor, and the standalone image, and stops each with `docker stop`. It fails if a container is SIGKILLed or takes close to the stop timeout, and checks that the executor shut down gracefully. Closes apache#2589.
The `exec` and tini changes can be checked by reading them, so a CI job that starts containers and depends on log lines and timing costs more than it adds. The Docker guide's stop section described expected behaviour. The Kubernetes guide keeps the part that isn't obvious: the pod's termination grace period and the scheduler's executor grace period both cap the drain at 30 seconds by default.
Each binary is to handle SIGTERM itself, so it stops on the signal as PID 1, whether the container runs the image's entrypoint or another command. The entrypoints' exec is then enough, and the images don't need tini.
The scheduler installed no signal handlers. Outside a container, SIGTERM's default action stopped it. As PID 1 in a container that default doesn't apply, so `docker stop` and the kubelet waited out the grace period to SIGKILL it. The binary now exits on SIGTERM or Ctrl-C itself. It doesn't drain its connections yet (apache#565).
As PID 1 in a container, SIGTERM's default action doesn't apply, so `docker stop` waited out its timeout to SIGKILL the CLI. The CLI now exits on SIGTERM itself, with the status the default action would give. Ctrl-C is left to the REPL.
The scheduler had a copy of the result service's `shutdown_signal`. It now lives in `ballista_core::utils`, which both use. `ballista_result_service::shutdown_signal` re-exports it.
villebro
force-pushed
the
fix/docker-entrypoints-sigterm
branch
from
October 14, 2026 17:59
0d3dd03 to
af53a26
Compare
…SIGTERM The SIGTERM handler exits straight away, which skipped the REPL's history save at the end of its loop and the TUI's terminal restore. The REPL now writes each history entry as it's added, so the history survives any stop. The handler leaves the TUI's raw mode and alternate screen before exiting.
villebro
marked this pull request as ready for review
October 14, 2026 22:08
Member
|
Thanks @villebro, and thank you @avantgardnerio for reviewing |
coderfender
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
With this change, containers built from the published images stop the way their Ballista processes expect. Today each entrypoint runs its binary as a child of bash, which stays PID 1 and drops the SIGTERM from
docker stopor the kubelet. Executors never drain, and every container waits out its grace period to be SIGKILLed.Closes #2589.
What changes are included in this PR?
exectheir binary, so it runs as PID 1 and receives SIGTERM, as the result service's entrypoint already does.ballista_core::utils::shutdown_signal.ballista_result_service::shutdown_signalre-exports it./dev/tcp, because the image doesn't installnc.init: truefor itssleep infinity.terminationGracePeriodSecondsand the scheduler's--executor-termination-grace-period. Raise both if tasks can run longer.Related: #565 (the scheduler now exits on SIGTERM; draining is still to come), #2357 (with #2448, executor pods also drain their Flight streams), #2525.
Are there any user-facing changes?
Yes, all in
docs/source/upgrading/56.0.0.md:command:, behave the same.How was this tested?
shellcheckpasses on every changed script..history. Normal\qexits across sessions keep each entry once.ubuntu:26.04base and entrypoint shape. Withoutexec,docker stopnever reaches the process, which is SIGKILLed (exit 137). Withexec, a process that handles SIGTERM gets it and stops straight away.