feat: enable migration of key-shares into a populated keyshare storage

[barakeinav1]

Background

Latest testing
of the migration flow revealed that migrating node A → node B → back to node A does not work out of the box.
The second migration (back to the original node) appears to succeed at the backup-cli and TLS layer, but the MPC contract state does not update and the migration does not complete without manual intervention (see full reproduction steps and workaround in the link).

This flow is useful in at least the following scenarios:

An operator wants to keep an operational (dormant) backup node at all times and occasionally fail back to the original node.

An operator wants to reuse the original node after migration (e.g., to avoid re-provisioning and setup time for a fresh node).

Request: the migration flows (MPC node, CLI) supports migration to an existing old node.

User Story

No response

Acceptance Criteria

migration flow is updates and tested

Resources & Additional Notes

Please note #2121 (comment)

[netrome]

Thanks for raising this! Setting this as P1, let's prioritize this for next week.

[barakeinav1]

Did some research with Claude on this. Sharing the hypotheses for the record — not validated against logs from an actual repro.

Likely two-step failure

1. Node skips the put-keyshares wait when disk already has matching keyshares. In crates/node/src/migration_service/onboarding.rs → execute_onboarding, the if get_keyshares(&importing_keyset).is_err() early-return causes the destination to bypass wait_for_and_import_keyshares — exactly what happens in a back-migration, since the keyset doesn't change. The node sends conclude_node_migration immediately based on old on-disk state, without any operator-driven sync.

2. Stale TEE attestation makes the contract reject conclude_node_migration. Contract-side TEE re-verify at crates/contract/src/lib.rs:2289-2306 requires a valid attestation; PCCS collateral is 7 days. If the destination was offline between migrations, its get_tee_accounts entry is from before the previous migration. periodic_attestation_submission runs on an interval, so the node may retry conclude with a still-stale on-chain attestation.

"delete + restart" fixes both: wiping disk forces the node back into wait_for_and_import_keyshares, and restart kicks periodic_attestation_submission early so a fresh attestation lands before the next conclude attempt.

If the attestation wasn't actually stale, other candidates

• Wrong signer_account_pk was passed to start_node_migration. The destination can't sign conclude with a key the contract doesn't expect; tx fails silently from the operator's POV (expected_signer_pk == signer_pk check at crates/contract/src/lib.rs:2283-2289).
• Destination's blockchain indexer lags the migration_info update (similar shape to node(indexer): mpc_indexer_latest_block_height can stall at startup while near_block_height_head advances normally #3295) — active_migration never flips true on the node, so OnboardingJob::Onboard is never produced and conclude_node_migration is never sent.

docker logs mpc-node on the destination during a failed back-migration would distinguish these — it'd show whether OnboardingJob::Onboard was entered, whether the conclude tx was sent, and if sent, the contract's rejection error.

[barakeinav1]

Thanks for raising this! Setting this as P1, let's prioritize this for next week.

think a bit about the use case:
-> keeping 2 up to date valid CVMs, so if one falls you can switch back and forth - valid use case.
-> after a real upgrade-> old CVM may be still valid for some time. so keeping it as an operational backup can be useful.

[barakeinav1]

Reproduction update — 2026-05-24

Both reproduction attempts pass — A → B → A finalizes cleanly with the cluster signing afterwards. That rules out the P1 early-return as load-bearing and pushes the remaining weight onto P2 (stale on-chain attestation).

Test 1: Rust E2E harness (mocked attestation)

Forked migration_service__should_migrate_nodes_via_backup_cli into migration_service__should_handle_back_migration_a_to_b_to_a (crates/e2e-tests/tests/migration_service.rs). Setup: 2 participants + 1 migration target (A0, A1, B0). Flow: A0 → B0 forward; kill_nodes([A0]) + start_nodes([A0]) (preserves keyshares on disk, the exact P1 trigger condition); B0 → A0 back. The contract's migration_info clears in both directions, participant TLS keys update correctly, and a sign request succeeds after each direction. ~36 s total. Attestation throughout is {"Mock": "Valid"}, which is the harness's only sidestep.

Test 2: Real-TDX localnet (real Dstack attestation)

New driver: localnet/tee/scripts/rust-launcher/test-migration.sh. Three real TDX CVMs (A0, A1, B0). B0 shares A0's NEAR account but has a distinct P2P key, mirroring the production migration scenario. Full A → B → A round-trip:

[PASS] Forward migration A0 -> B0 succeeded
[PASS] Back-migration B0 -> A0 succeeded (bug NOT reproduced on real-TDX localnet)

A0 was killed with vmm stop -f and brought back with vmm start between the two directions — state preserved, keyshares on disk. Attestations were real Dstack (verified by the contract via PCCS collateral). End state: node0 participant TLS key reverted to A0's, migration_info.destination_node_info: null, ECDSA sign works.

What this means for the original hypothesis

• P1 (early-return in execute_onboarding) is ruled out as load-bearing. Both tests exercise the exact disk-state precondition and the back-migration still finalizes. P1 remains an architectural smell — concluding without operator sync — but it doesn't break the production path on its own.
• P2 (stale on-chain tee_state entry rejected by reverify_participants) is the only remaining candidate that fits the symptom. The Rust E2E sidesteps P2 via mocked attestation. The real-TDX localnet sidesteps P2 by accident: the whole test runs in minutes, well within PCCS collateral TTL (~7 days), so A0's on-chain attestation is still fresh even after the stop/start. The production failure happened after A was offline long enough for that to age out.

This also fits the earlier operator pushback that "the attestation wasn't stale" — what's stale in P2 is the on-chain tee_state entry (last submit_participant_info for that NodeId), not any artifact visible at attestation-creation time.

Next concrete step

Contract-sandbox test that puts the contract in Running with get_tee_accounts entries pre-dating the PCCS collateral window, then calls start_node_migration + conclude_node_migration directly and asserts InvalidParameters::InvalidTeeRemoteAttestation. If that reproduces the on-chain failure, the operator playbook fix is to require an attestation refresh (extra submit_participant_info on startup, or operator-driven before conclude_node_migration).

Side findings while wiring up the localnet TEE test

The localnet TEE scripts were missing the port plumbing for backup-cli to reach the MPC node's migration endpoint. Now fixed on the branch:

• node.conf.{localnet,testnet}.toml.tpl: migration_web_ui 8078 → 8079 (matches production).
• deploy-tee-cluster.sh: PORTS adds 8079:8079.
• deploy-launcher.sh: optional EXTERNAL_MPC_MIGRATION_HTTP_PORT / INTERNAL_MPC_MIGRATION_HTTP_PORT drive the host→CVM QEMU forward (backward compatible).