Database sync does not recreate a Replicated database on a replica rebuilt from an empty PVC (code 253 on the stale `<shard>|<replica>` registration; skipped entirely while the replica is NotReady)

[YafetMelake]

Environment

• clickhouse-operator v0.0.7, Helm chart clickhouse-operator-helm 0.0.7, image digest sha256:d51ca53f09677ebf46ac78bddf8d2dc5e0e412bd2062be3e596bed5879748886
• CRDs clickhouse.com/v1alpha1 ClickHouseCluster and KeeperCluster
• ClickHouse server 26.8.5.13, ClickHouse Keeper 26.8.5.13
• kind v0.32.0, Kubernetes v1.36.1 (1 control-plane, 3 workers)
• ClickHouseCluster: 1 shard x 2 replicas, spec.settings.enableDatabaseSync: true; KeeperCluster: 3 nodes
• One Replicated database (ENGINE = Replicated('/clickhouse/databases/<db>', '{shard}', '{replica}'), macros are the operator's numeric ones, so the replica registrations are 0|0 and 0|1) holding six ReplicatedReplacingMergeTree tables with a few million rows

Steps to reproduce

1. Install a ClickHouseCluster with 2 replicas and a 3-node KeeperCluster, database sync enabled.
2. Create a Replicated database on the cluster and a few Replicated tables in it, insert data, confirm both replicas hold the same tables and rows.
3. Discard replica 1's volume and pod: kubectl delete pvc clickhouse-storage-volume-<cluster>-clickhouse-0-1-0 --wait=false then kubectl delete pod <cluster>-clickhouse-0-1-0 --wait=false.
4. Watch the StatefulSet recreate the pod and PVC, then watch the operator log and kubectl get chc <cluster> -o jsonpath='{range .status.conditions[*]}{.type}={.status}/{.reason}{"\n"}{end}'.

Run four times on 2026-09-18: twice with the operator's default readiness probe (HTTP /ping on 9002), twice with a custom containerTemplate.readinessProbe (an exec probe that fails a replica whose system.tables count for the database trails its peers'). The operator never recreated the database in any pass; the mechanism differs between the two probes.

Observed with the default readiness probe (2 passes)

• The StatefulSet recreated the pod object after 8 s and the PVC after 1 s; the container was Ready after 15 s and the pod Ready 3 s later, holding 0 tables and already listed as an endpoint of the cluster's client Service. There is no readiness gate in 0.0.7, so an empty replica takes traffic.

• Database sync detected the missing database and issued its same-UUID CREATE DATABASE IF NOT EXISTS <db> UUID '<uuid>' ENGINE = Replicated(...) on the new replica every 5 s. Every attempt failed with:

  Code: 253. DB::Exception: Replica node '/clickhouse/databases/<db>/replicas/0|1/digest' in ZooKeeper already exists and contains unexpected value

  The condition read SchemaInSync=False/DatabasesNotCreated while Healthy stayed True. In pass 1 this went on until we intervened (see workaround); the operator never dropped the stale registration because the stale replica is the one it is trying to recreate.

• Pass 2, dropping only the database registration by hand (SYSTEM DROP DATABASE REPLICA '0|1' FROM DATABASE <db> on the surviving replica): the operator's CREATE succeeded 5 s later, then its SYSTEM SYNC DATABASE REPLICA <db> hung. The DDLWorker on the new replica logged, every 5 s:

  Code: 253. DB::Exception: Replica /clickhouse/tables/<table uuid>/0/replicas/1 already exists

  because the tables' own replica znodes from the lost volume were still there. A DROP DATABASE <db> SYNC on the new replica blocked behind the hung sync and hit max_execution_time (code 159) after 120 s.

• After a second SYSTEM DROP DATABASE REPLICA '0|1' (issued while the new replica already held the recreated database) the database became a zombie: it exists locally, the DDLWorker logs '/clickhouse/databases/<db>/replicas/0|1/log_ptr': node doesn't exist, DROP DATABASE ... SYNC fails with KEEPER_EXCEPTION No node, and the operator reports SchemaInSync=True/ReplicasInSync although the replica holds 0 of 6 tables. The condition compares databases, not tables.

• No operator metric or event distinguishes this from a healthy cluster; only the operator log and the per-host table counts do.

Observed with a readiness probe that holds the empty replica NotReady (2 passes)

• The StatefulSet recreated the pod object after 6-12 s and the PVC after 0-6 s; the container was running after 6-9 s. The probe kept the pod NotReady (READY 0/1, ready=false in the client Service's EndpointSlice) for the whole observation window (100 s in one pass, 60 s in the other).
• The operator never issued the CREATE: 0 lines with code 253 in the window. Every reconcile logged no replicas to replicate schema, skipping, because the sync loads a replica's state only while the pod's ContainersReady condition is True (internal/controller/clickhouse/sync.go around line 515 at main commit a58f9b3; the same check is in v0.0.7) and skips the step with fewer than two queryable replicas (around line 883 at that commit). A user-supplied container readiness probe therefore starves the sync on exactly the replica that needs it; a pod readiness gate would not, since ContainersReady ignores gates.
• Conditions during the hold: Healthy=False/ReplicasNotReady, SchemaInSync=False/DatabasesNotCreated, Ready=True/AllShardsReady.
• The sync ran (and found nothing to do) only after our manual recovery had made the pod Ready.

Expected

Any of:

1. Database sync treats a replica whose registration exists in Keeper but whose local data directory is empty like a new replica: drop the stale <shard>|<replica> database registration and the tables' <replica> registrations for that shard/replica pair (SYSTEM DROP DATABASE REPLICA ... FROM DATABASE, SYSTEM DROP REPLICA '<replica>' FROM DATABASE <db> on a peer, or SYSTEM RESTORE DATABASE REPLICA on the rebuilt host), then run its CREATE and SYNC as it does for a scaled-up replica.
2. The operator surfaces the state clearly: a condition or event that says the replica cannot be initialised because of a stale registration, SchemaInSync reporting False while table counts differ, and the replica held out of the Service until the sync succeeds.
3. The sync reaches a replica whose container is running but NotReady (the management port answers regardless of readiness), so that a readiness probe which waits for the schema does not prevent the schema from arriving.

Workaround used

On the surviving replica (SYSTEM DROP REPLICA cannot drop the local replica), then on the rebuilt one:

-- on the peer
SYSTEM DROP DATABASE REPLICA '0|1' FROM DATABASE <db>;
SYSTEM DROP REPLICA '1' FROM DATABASE <db>;
-- on the rebuilt replica: the operator's own statement, with engine_full read from the peer
CREATE DATABASE IF NOT EXISTS <db> UUID '<uuid>' ENGINE = Replicated('/clickhouse/databases/<db>', '{shard}', '{replica}');

All six tables appeared 2-6 s after the drops and row counts matched at once (117-162 s from the PVC delete to parity, most of it our own wait for the operator). If the operator's CREATE has already succeeded and its SYNC is hanging, KILL QUERY WHERE user = 'operator' SYNC followed by SYSTEM RESTORE DATABASE REPLICA <db> on the rebuilt replica recovered the database in 3 s. We also run an exec readiness probe that compares the local user-table count with the peers' through clusterAllReplicas, so an empty replica stays NotReady until it is rebuilt; with the current sync logic that probe also means the rebuild is always manual.