mpc-node: handle SIGTERM for graceful shutdown on operator stop

[barakeinav1]

Background

mpc-node does not install a SIGTERM handler today. When an operator stops the process via docker stop, kubectl delete, systemctl stop, or dstack's CVM stop command, the orchestrator sends SIGTERM first and then SIGKILL after a grace period (10s for Docker default, 30s for Kubernetes, 90s for systemd). Because we have no handler installed, SIGTERM has the same effect as SIGKILL — the OS terminates the process immediately, the embedded near-indexer thread is killed mid-write, and the next start can land on inconsistent RocksDB state.

This issue was surfaced while investigating docs/investigation/2121-back-migration-e2e-flake.md, where a CI test SIGKILLs mpc-node mid-flight and the next start panics ~65–80% of the time inside near-indexer (see docs/investigation/nearcore-indexer-sigkill-restart-panic.md). Production stops via dstack/Docker/Kubernetes/systemd take the same code path, so any production stop today carries the same restart-corruption risk as the test scenario.

User Story

As an operator stopping an MPC node via my orchestrator (dstack CVM stop / Docker / Kubernetes / systemd), I want the node to receive SIGTERM, finish in-flight commits, and exit cleanly within the grace period — so that the next start finds RocksDB in a consistent state and doesn't trip an indexer restart panic.

Acceptance Criteria

• mpc-node installs a SIGTERM handler that routes the signal into the existing internal shutdown channel (shutdown_signal_sender), so the same tokio::select! arm that handles TEE image-hash shutdowns also handles SIGTERM.
• After the main select! exits, near_async::shutdown_all_actors() is called so nearcore's actor system can commit any in-flight RocksDB batches before the process exits.
• tracing::warn!("SIGTERM received, initiating graceful shutdown") is emitted when the signal arrives, so operators can confirm the path was taken.
• Verified in CI: mpc-node exits gracefully (typically within 100 ms) after SIGTERM, vs SIGKILL fallback firing in the pre-fix state.

Resources & Additional Notes

• The investigation that surfaced this issue and the test campaign data live in docs/investigation/2121-back-migration-e2e-flake.md. With a working handler, the e2e test passed 1/5 vs 0/5 without it on the same commit — confirming the handler is a real production improvement but does not fully close the upstream nearcore restart panic, which fires non-deterministically regardless of shutdown cleanliness.
• The upstream nearcore bug is documented separately in docs/investigation/nearcore-indexer-sigkill-restart-panic.md. That issue needs to be fixed in nearcore; this issue is the orthogonal mpc-node-side fix that should land regardless.
• We considered also calling near_store::db::RocksDB::block_until_all_instances_are_dropped() (which neard's standalone binary does after shutdown_all_actors), but it hangs indefinitely in our embedding because our indexer thread's block_on never returns — the spawned monitor tasks hold Arc<IndexerState> → Arc<RocksDB> references that nothing currently cancels on shutdown. A proper fix for that hang would wire a CancellationToken through the indexer thread; out of scope for this issue.

[barakeinav1]

Scope clarification: what this handler shuts down, and what it doesn't

After running a 5-cycle CI campaign on the handler (PR #3410) we know exactly what's graceful and what isn't, so worth recording it on this issue too:

What the handler covers (the "100 ms graceful shutdown" the diagnostic logs):

• mpc-node's main runtime path: SIGTERM → existing internal shutdown channel → main select! exit → cancellation_token.cancel() → image-hash watcher join.
• A stop signal to nearcore's actor system via near_async::shutdown_all_actors(). This call is synchronous — it signals stop and returns; it does not wait for actors to finish.

What the handler does NOT cover:

• The embedded near-indexer thread. It runs in a separate std::thread::spawn'd closure with its own tokio runtime (crates/node/src/indexer/real.rs:80). That runtime hosts the near_indexer::streamer::start block-streaming loop plus ~6 tokio::spawn'd monitor tasks (indexer_logger, monitor_allowed_docker_images, monitor_allowed_launcher_compose_hashes, monitor_tee_accounts, monitor_allowed_foreign_chain_providers, foreign_chain_whitelist_verifier::run). Each holds Arc<IndexerState> → transitively Arc<RocksDB>.
• Nothing in our handler propagates a cancellation into that runtime, so those tasks keep running until the std::thread is terminated by OS-level process exit (~100 ms after SIGTERM).

The 100 ms timing in the diagnostic is itself proof the indexer didn't actually finish anything — a real near-indexer drain takes seconds, not milliseconds.

We initially tried to wait for it: the first iteration of the handler also called near_store::db::RocksDB::block_until_all_instances_are_dropped() (which is what neard's standalone binary does next). It hung indefinitely in 5/5 CI runs because the spawned monitor tasks held Arc refs that nothing was cancelling — so the refcount never reached zero. The 60 s grace expired and the orchestrator SIGKILLed us, landing back at the uncommitted-RocksDB state the handler was supposed to prevent. We removed the call (see the inline comment in run.rs).

What a "fully graceful indexer shutdown" would require (out of scope for this issue):

Wire a CancellationToken through spawn_real_indexer, have each spawned task select on it, and have the outer indexer-thread closure await its tasks before returning. Then block_on would actually return, the Arc<IndexerState> would drop, RocksDB refs would drop, and block_until_all_instances_are_dropped could safely run. Tracked as a follow-up; not necessary for the production improvement this issue/PR delivers (operators now get a real graceful main-runtime shutdown on dstack/Docker/K8s/systemd stop, vs SIGTERM-equivalent-to-SIGKILL before).

Full investigation context including the campaign data is in docs/investigation/2121-back-migration-e2e-flake.md (sections "Real SIGTERM handler in mpc-node — also does not fix it" and the iteration-1/iteration-2 subsections) and the upstream-facing summary is in docs/investigation/nearcore-indexer-sigkill-restart-panic.md (section "Strongest evidence: actor-system shutdown doesn't help" with the explicit caveat).