[ROMEO-1265] Exit cleanly on SIGTERM received during boot

[ivan-zakharanka-airhelp]

Issue

Workers that are scaled to zero get resurrected by a deploy (the chart applies replicas: 1), then the autoscaler's 30s probe sees an empty queue and scales them back down. The SIGTERM therefore lands at pod age ~10–20s — while the app is still booting.

CLI#run boots the application before any signal handling exists:

def run
  boot_system                 # loading Rails takes ~17s in our app
  Eventboss.logger.info('Starting eventboss...')
  Eventboss.launch            # Runner#setup_signals installs traps in here
end

A SIGTERM arriving in that window hits no handler at all, so the process dies from the default disposition. bin/eventboss wraps everything in rescue => e ... exit 1, so the orchestrator sees a non-zero exit and reports a crash — for us that means an SMS page for what is actually an ordinary, harmless scale-down.

This is not specific to the autoscaler: any mid-boot SIGTERM does it (Spot reclaim, node drain, rebalancing, kubectl rollout restart).

Why not copy this from Sidekiq

It was suggested that since eventboss borrowed its signal handling from Sidekiq, we could transplant Sidekiq's solution. Sidekiq has the same gap — lib/sidekiq/cli.rb#L42-L68: boot_application is line 43, the IO.pipe + Signal.trap loop only starts at line 50. Sidekiq just doesn't get bitten in practice, because its workers aren't usually scaled to zero seconds after starting. The self-pipe pattern eventboss already copied into Runner#setup_signals is the Sidekiq code — same code, same place relative to boot.

Solution

Install a minimal SIGTERM trap before booting:

def run
  setup_boot_signal_traps
  boot_system
  ...
  Eventboss.launch
end

def setup_boot_signal_traps
  Signal.trap('SIGTERM') { exit 0 }
end

This only covers the boot window. Once boot completes, Runner#setup_signals overwrites the trap with the normal graceful shutdown (drain the launcher, then exit 0), so runtime behaviour is unchanged — no in-flight message can be lost by this. There is nothing to drain during boot anyway: no listeners are running, so no SQS message has been received yet. Eventboss deletes messages only after successful processing, so anything mid-flight simply returns to the queue.

The trap is deliberately minimal rather than a second self-pipe: it exists purely so the exit code is 0 instead of a signal death.

Testing

New spec forks a process, stubs boot_system to block, sends SIGTERM mid-boot and asserts the exit status:

• before the fix: exitstatus is nil (killed by signal, no clean exit)
• after the fix: exitstatus is 0

Full suite green (109 examples, 0 failures) on Ruby 3.4.5.

Notes

• Version bumped to 1.9.9 + CHANGELOG entry, so consumers can pick this up.
• Follow-up worth considering separately: Sidekiq chains the previous handler rather than clobbering it (cli.rb#L55-L65), while Runner#setup_signals replaces traps unconditionally. Not needed for this bug, but it would let a host app keep its own TERM handling.
• Complementary to, not a replacement for, the chart-side change that stops resurrecting scaled-to-zero workers on deploy. That removes most of the occurrences; this makes the process correct whenever a mid-boot SIGTERM does happen.

🤖 Generated with Claude Code