Skip to content

[v92-audit] atomic-slot-migration: RDB slot-import opcode is loaded with cluster-enabled no, crashing standalone servers and standalone replicas of cluster primaries #76

Description

@madolson

rdb.c:3661 dispatches RDB_OPCODE_SLOT_IMPORT to clusterRDBLoadSlotImport() without checking server.cluster_enabled, unlike the adjacent RDB_OPCODE_SLOT_INFO case at rdb.c:3655 which does. When a server with cluster-enabled no loads an RDB that carries an in-progress slot import, createSlotImportJob() marks the ranges importing in a kvstore that has only one hashtable, so kvstoreSetIsImporting() fails assert(didx >= 0 && didx < kvs->num_hashtables) at kvstore.c:949 for any slot above 0, and if the range is slot 0 only it gets past that and dereferences the NULL server.cluster at cluster_migrateslots.c:427.

No crafted file is needed. A plain REPLICAOF from a non-cluster node to a cluster-mode primary that has a slot import in flight crashes the replica on full sync, and it crashes again on every retry for as long as the import is in flight. Restarting a cluster node's own data directory with cluster-enabled no crashes at startup on the dump.rdb the node wrote itself.

Details

Repro 1: standalone replica full-syncs from a cluster primary mid-import

Two cluster primaries, 7560 and 7561, 7561 owns 0-199. Pin the target so the import stays in flight:

$ valkey-cli -p 7560 debug slotmigration prevent-failover 1
OK
$ valkey-cli -p 7561 cluster migrateslots slotsrange 0 199 node <7560-id>
OK
$ valkey-cli -p 7560 cluster getslotmigrations | grep -A1 state
state
failover-granted

Now start a non-cluster server as its replica:

$ valkey-server --port 7574 --dir /tmp/asmplain --save '' --replicaof '127.0.0.1 7560'
$ valkey-cli -p 7574 ping
Could not connect to Valkey at 127.0.0.1:7574: Connection refused

Replica log, verbatim:

20473:S 16 Sep 2026 14:38:01.316 * Full resync from primary: b9c207140d4017d8941108b944854d44c2e83537:1827803
20473:S 16 Sep 2026 14:38:02.325 # === ASSERTION FAILED ===
20473:S 16 Sep 2026 14:38:02.325 # ==> kvstore.c:949 'didx >= 0 && didx < kvs->num_hashtables' is not true

------ STACK TRACE ------

20473 valkey-server *
/home/matolson/v92audit/build-plain/src/valkey-server *:7574[0x493d84]
/home/matolson/v92audit/build-plain/src/valkey-server *:7574(clusterRDBLoadSlotImport+0x1ca)[0x495e6a]
/home/matolson/v92audit/build-plain/src/valkey-server *:7574(rdbLoadRioWithLoadingCtx+0x89d)[0x5b441d]
/home/matolson/v92audit/build-plain/src/valkey-server *:7574(rdbLoad+0x18c)[0x5b618c]
/home/matolson/v92audit/build-plain/src/valkey-server *:7574(replicaLoadPrimaryRDBFromDisk+0x11b)[0x5b64fb]
/home/matolson/v92audit/build-plain/src/valkey-server *:7574(handleBioThreadFinishedRDBDownload+0xea)[0x56017a]
/home/matolson/v92audit/build-plain/src/valkey-server *:7574(replicationCron+0x150)[0x560e30]
/home/matolson/v92audit/build-plain/src/valkey-server *:7574(serverCron+0x6e0)[0x549af0]

Same run under AddressSanitizer, which resolves the frames:

==13085==ERROR: AddressSanitizer: global-buffer-overflow on address 0x000000c78018 at pc 0x7f5bc911821a
    #1 0x5c945e in memtest_preserving_test src/memtest.c:304
    #2 0x54d156 in memtest_test_linux_anonymous_maps src/debug.c:2290
    #3 0x54d3db in doFastMemoryTest src/debug.c:2333
    #4 0x54f38f in printCrashReport src/debug.c:2537
    #5 0x54e289 in _serverAssert src/debug.c:1182
    #6 0x5b1334 in kvstoreSetIsImporting src/kvstore.c:949
    #7 0x50446f in setSlotImportingStateInDb src/cluster_migrateslots.c:246
    #8 0x50460e in setSlotImportingStateInAllDbs src/cluster_migrateslots.c:257
    #9 0x5047d7 in createSlotImportJob src/cluster_migrateslots.c:835
    #10 0x50676f in clusterRDBLoadSlotImport src/cluster_migrateslots.c:426
    #11 0x688ef2 in rdbLoadRioWithLoadingCtx src/rdb.c:3662
    #12 0x68b0f2 in rdbLoadRioWithLoadingCtxScopedRdb src/rdb.c:3522
    #13 0x68b234 in rdbLoadRio src/rdb.c:3511
    #14 0x68b576 in rdbLoad src/rdb.c:4073
    #15 0x6a61fa in replicaLoadPrimaryRDBFromDisk src/replication.c:2951
    #16 0x6b456c in handleBioThreadFinishedRDBDownload src/replication.c:5724

The reported global-buffer-overflow is the crash reporter's own memtest, not the defect. The defect is frame #6.

Repro 2: a cluster node's own dump.rdb, restarted with cluster-enabled no

With the same in-flight import on 7560:

$ valkey-cli -p 7560 save
OK
$ cp n7560/dump.rdb /tmp/asmrdb2/dump.rdb
$ valkey-server --port 7571 --dir /tmp/asmrdb2 --save ''
/home/matolson/v92audit/build-plain/src/valkey-server *:7571(clusterRDBLoadSlotImport+0x1ca)[0x495e6a]

python3 -c "d=open('n7560/dump.rdb','rb').read(); print(d[:9], d.count(b'\xf3'))" gives b'VALKEY080' 2, so the opcode really is in the file the server wrote.

Repro 3: the slot-0-only case, which reaches the NULL deref instead of the assert

63-byte hand-built RDB, one import job, range 0-0:

b = b'VALKEY080' + bytes([243]) + bytes([40]) + b'a'*40 + bytes([1,0,0]) + bytes([255]) + b'\x00'*8
open('/tmp/asmrdb/dump.rdb','wb').write(b)
25633:M 16 Sep 2026 14:32:43.433 * New slot import job created: {name: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, operation: import, source_node_id: unknown, slots: 0-0}.
ASAN:DEADLYSIGNAL
==25633==ERROR: AddressSanitizer: SEGV on unknown address 0x000000020050 (pc 0x0000005067a4 ...)
==25633==The signal is caused by a READ memory access.
    #0 0x5067a3 in clusterRDBLoadSlotImport src/cluster_migrateslots.c:427
    #1 0x688ef2 in rdbLoadRioWithLoadingCtx src/rdb.c:3662
    #2 0x68b0f2 in rdbLoadRioWithLoadingCtxScopedRdb src/rdb.c:3522
    #3 0x68b234 in rdbLoadRio src/rdb.c:3511
    #4 0x68b576 in rdbLoad src/rdb.c:4073
    #5 0x71b783 in loadDataFromDisk src/server.c:7696
    #6 0x71ce10 in main src/server.c:8196
SUMMARY: AddressSanitizer: SEGV src/cluster_migrateslots.c:427 in clusterRDBLoadSlotImport

0x20050 is offsetof(clusterState, slot_migration_jobs) off a NULL server.cluster. Plain build, same input:

29299:M 16 Sep 2026 14:32:57.849 # valkey 255.255.255 crashed by signal: 11, si_code: 1
29299:M 16 Sep 2026 14:32:57.849 # Accessing address: 0x20050
/home/matolson/v92audit/build-plain/src/valkey-server *:7570(clusterRDBLoadSlotImport+0x1d4)[0x495e74]

Why

server.cluster is only allocated when server.cluster_enabled (server.c:8160-8162 calls clusterInit() under that condition), and in standalone mode every kvstore has one hashtable rather than 16384. The RDB dispatch does not test either:

  • src/rdb.c:3655 for RDB_OPCODE_SLOT_INFO: if (server.cluster_enabled && slot_id < CLUSTER_SLOTS) {
  • src/rdb.c:3661 for RDB_OPCODE_SLOT_IMPORT: no guard at all.

clusterRDBLoadSlotImport validates the name length and the slot bounds (src/cluster_migrateslots.c:406-419, hardening from #4210 and #4229) but never checks that cluster mode is on before calling createSlotImportJob at src/cluster_migrateslots.c:426 and listAddNodeTail(server.cluster->slot_migration_jobs, ...) at src/cluster_migrateslots.c:427.

Introduced by 383ffe4 (#2635), which added the rdb.c dispatch.

The fix that matches the neighbouring case is to skip the opcode's payload when !server.cluster_enabled rather than to return C_ERR, since C_ERR becomes goto eoferr at src/rdb.c:3662 and would turn the crash into a refusal to load any data at all. #4207 and #4222 are the two prior hardening reports on this loader; both were filed against cluster-enabled yes and neither covers the cluster-disabled case.

Testing

Reproduced on commit 3f9062e on both a plain make build and a SANITIZER=address build. All three repros above are from that commit. Repro 1 needs debug slotmigration prevent-failover 1 on the target only to widen the window; without it the same crash happens whenever a full sync happens to overlap a migration, which is what makes it more than a crafted-file bug.

This was generated by AI but verified, with love, by a human.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions