Skip to content

Implement graceful shutdown for web server on SIGTERM/SIGINT - #207

Open
Just-Bamford wants to merge 2 commits into
Vero-protocol:mainfrom
Just-Bamford:fix/issue--server-graceful-shutdown
Open

Implement graceful shutdown for web server on SIGTERM/SIGINT#207
Just-Bamford wants to merge 2 commits into
Vero-protocol:mainfrom
Just-Bamford:fix/issue--server-graceful-shutdown

Conversation

@Just-Bamford

@Just-Bamford Just-Bamford commented Aug 21, 2026

Copy link
Copy Markdown

Description

Problem

The web server in index.js was not implementing graceful shutdown handlers for SIGTERM/SIGINT signals. During rolling deploys or container restarts:

  • In-flight webhook requests were abruptly terminated without draining
  • The database connection pool was forcefully closed without cleanup
  • The config poller interval/worker was left running without proper teardown

Closes #199

This created data integrity risks and violated deploy best practices. The event worker (src/workers/event-worker.js) already had the correct graceful shutdown pattern implemented (lines 182-211), but this safety mechanism was missing from the web server process.

Solution

Applied the existing graceful shutdown pattern from event-worker.js to the web server's startServer() function in index.js. The implementation now:

  1. Registers signal handlers for SIGTERM and SIGINT in the startServer() function
  2. Stops accepting new connections via server.close()
  3. Allows in-flight requests to drain before proceeding
  4. Closes the config poller (stops interval or terminates worker thread)
  5. Gracefully shuts down the database pool (releases all connections)
  6. Exits cleanly with process.exit(0)
  7. Forces exit after 30 seconds if graceful shutdown stalls

Changes Made

index.js

  • Added import for stopConfigPoller from ./src/services/config-poller
  • Updated import for src/db/client.js to include the shutdown function (aliased as shutdownDbPool)
  • Implemented async shutdown(signal) handler within startServer() that:
    • Prevents concurrent shutdown calls with a closing flag
    • Logs shutdown initiation with the signal type
    • Calls server.close() to stop accepting new connections
    • Drains and closes config poller
    • Drains and closes database pool
    • Sets 30-second timeout to force exit if graceful shutdown hangs
    • Handles errors during shutdown gracefully
  • Registered SIGTERM and SIGINT handlers that invoke the shutdown function

src/db/client.js

  • Removed automatic SIGTERM/SIGINT handlers that were bypassing graceful shutdown
  • The shutdown() function is now explicitly called by the server's shutdown handler rather than being triggered implicitly
  • This ensures coordinated shutdown: server drains → config poller stops → db pool closes → exit

Behavior

Before: Container receives SIGTERM → process exits immediately → in-flight requests dropped, pool closes abruptly

After: Container receives SIGTERM → server stops accepting new connections → existing requests complete → config poller stops → db pool drains cleanly → process exits with code 0

Verification

  • All existing unit tests pass (32 tests pass, 24 skip due to missing test database - unrelated to this change)
  • No syntax errors in modified files
  • Implementation mirrors the proven pattern already in use by event-worker.js
  • Graceful shutdown is non-blocking and handles errors gracefully

Acceptance Criteria Met

✓ SIGTERM sent to running server drains in-flight requests before exiting
✓ Database pool closes cleanly
✓ Config poller stops cleanly
✓ Behavior mirrors event-worker.js shutdown pattern
✓ 30-second timeout prevents infinite hangs
✓ Error logging for debugging shutdown issues

Comment thread index.js
}
app.post(
"/github-webhook",
ingestRateLimiter,
Comment thread index.js Fixed
@N-thnI

N-thnI commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

please ensure all checks pass

- Register SIGTERM/SIGINT handlers in index.js startServer()
- Stop accepting new connections via server.close()
- Drain in-flight requests before shutdown
- Close config poller interval/worker cleanly
- Close database pool gracefully with proper cleanup
- Set 30-second timeout to prevent infinite hangs
- Remove automatic shutdown handlers from db/client.js
- Mirrors proven pattern from event-worker.js

Fixes deploy/restart safety for rolling deployments and container restarts.
@Just-Bamford
Just-Bamford force-pushed the fix/issue--server-graceful-shutdown branch from 4b71697 to ab346a7 Compare August 23, 2026 10:06
@Just-Bamford

Copy link
Copy Markdown
Author

@N-thnI kindly review

@Just-Bamford

Just-Bamford commented Aug 30, 2026

Copy link
Copy Markdown
Author

@N-thnI haffa just close the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The Express web server has no graceful shutdown, unlike the worker process

3 participants