From c9c5631967cb07601954a840d6185235c48cf49e Mon Sep 17 00:00:00 2001 From: Niklas Schmidtmer Date: Wed, 15 Jul 2026 17:35:11 +0200 Subject: [PATCH 1/3] Revise rolling upgrade guide Mostly restructuring, separating step-by-step instructions from background knowledge about configuration parameters. --- docs/admin/upgrade/rolling.md | 396 +++++++++++++++++++--------------- 1 file changed, 224 insertions(+), 172 deletions(-) diff --git a/docs/admin/upgrade/rolling.md b/docs/admin/upgrade/rolling.md index 86292971..9036c324 100644 --- a/docs/admin/upgrade/rolling.md +++ b/docs/admin/upgrade/rolling.md @@ -1,118 +1,143 @@ -```{highlight} sh -``` - (rolling-upgrade)= # Rolling Upgrade ## Introduction -CrateDB provides an easy way to perform a rolling cluster upgrade with zero -downtime. +A rolling upgrade updates one CrateDB node at a time while the remaining nodes +continue serving requests. This minimizes service disruption and avoids +stopping the entire cluster. -A rolling upgrade is possible from one version to the next feature version. -Some examples: +For each node, decommission it gracefully, install the new version, +restart it, verify that it has rejoined the cluster, and then +continue with the next node. -- You can do a rolling upgrade from 4.5.x to 4.6.0 -- You can do a rolling upgrade from 4.8.x to 5.0.0 - (because 4.8 is the last feature release within the 4. series). -- You cannot do a rolling upgrade from x.y.z to x.(y + 3).z unless - the release notes explicitly mention support. +To perform a rolling upgrade of a cluster, stop one node at a time using the +{ref}`graceful stop ` procedure +(see {ref}`Signal Handling `). This will allow +CrateDB to relocate shards away from the node being decommissioned. +Depending on the configured availability mode, replica shards may temporarily +be unavailable and the cluster may become yellow. The node being decommissioned +rejects new requests while allowing pending requests to finish. -:::{WARNING} -Rolling upgrades are only possible if you are using a stable version of -CrateDB. If you are upgrading to a testing version you must perform a full -cluster restart. +:::{NOTE} +Due to the distributed execution of requests, some client requests might +fail during a rolling upgrade. -Check the {ref}`release notes ` for the -version you are upgrading to for any specific instructions that may override -this. +Please always have retry mechanisms in place for critical queries. ::: -To perform a rolling upgrade of a cluster, one node at a time has to be stopped -using the {ref}`graceful stop ` procedure -(see {ref}`Signal Handling `). +## Version requirements -This procedure will disable a node, which will cause it to reject any new -requests but will make sure that any pending requests are finished. +A rolling upgrade is possible in the following scenarios: -:::{NOTE} -Due to the distributed execution of requests, some client requests might -fail during a rolling upgrade. +- Between patch-level releases of the same minor version +- From one minor version to the next within the same major version +- From one major version to the next major version + +For example, you can do a rolling upgrade: + +- From 6.1.2 to 6.1.3 +- From 6.0.x to 6.1.y +- From 5.10.x to 6.0.0 + +You cannot do a rolling upgrade from x.y.z to x.(y + 3).z unless the release +notes explicitly mention support. + +:::{WARNING} +Check the {ref}`release notes ` for the +version you are upgrading to for any specific instructions that may override +this. ::: -This process will ensure a certain data availability. Which can either be -`none`, `primaries`, or `full` and can be configured using the -{ref}`SET ` statement. +## Graceful stop particularities -Using `full`, all shards currently located on the node will be moved to the -other nodes in order to stop gracefully. Using this setting the cluster will -stay `green` the whole time. +To initiate a graceful stop that behaves as described in the introduction +of this document, the {ref}`ALTER CLUSTER DECOMMISSION ` +statement must be used. -Using `primaries`, only the primaries will be moved to other nodes. Using -this setting means that the cluster will go into the `yellow` warning state -if a node that has been stopped contained replicas that are then unavailable. +Stopping a node via the `TERM` user signal (invoked via `Ctrl+C` or +`systemctl stop crate`) will cause a normal stop of CrateDB, **without** +going through the graceful stop procedure. -Using `none`, there is no data-availability guarantee. The node will stop, -possibly leaving the cluster in the critical `red` state if the node -contained a primary that has no replicas that can take over. +Depending on the size of your cluster, stopping a CrateDB node gracefully +might take a while. You might want to check your server logs to see if the +graceful stop process is progressing well. In case of an error or a timeout, +the node will stay up, signaling the error in its log files. -## Requirements +Using the default settings, the node will shut down by moving all primary shards +off the node first. This ensures that a primary copy of each shard remains +available before the node shuts down. However, the cluster health will most +likely turn yellow, because replicas that lived on that node will be missing. -### Full minimum data availability +Keep in mind that relocating shards might take some time depending on the +number of shards and their size. A timeout will occur after the duration configured as {ref}`cluster.graceful_stop.timeout `. +In case of a timeout, the stop process will abort and the cluster will +start distributing shards evenly again. If you want to force a stop after +the timeout, even if the relocation is not finished, you can set {ref}`cluster.graceful_stop.force ` +to `true`. -If the `full` minimum data availability is configured the cluster needs to -contain enough nodes to hold the number of replicas that are configured even if -one node is missing. +:::{WARNING} +A forced stop does not ensure the minimum data availability defined in the +settings and may result in temporary or even permanent loss of data! +::: -For example if there are only two nodes in a cluster and a table has one -replica configured the `graceful stop` procedure will not succeed and abort -as it won't be possible to relocate the replicas. +By default, only the graceful stop process considers the cluster settings +described at {ref}`graceful stop `. -If a table has a range configured as number of replicas this will take into -account the upper number of replicas. +## Upgrade process -With two nodes and 0-1 replicas, the `graceful stop` procedure will -abort. +To run the actual upgrade process, follow the steps outlined below in the designated order. -In short: for the `full` graceful stop to work the following has to be true: +::::::{stepper} -``` -number_of_nodes > max_number_of_replicas + 1 -``` +### Ensure green health -### Primaries minimum data availability +We assume that the cluster is in good health before starting the upgrade. +This means all shards are allocated and there are no failed health checks, +such as breached disk space watermarks. -If the `primaries` minimum data availability is used, take care that there -are still enough replicas in the cluster after a node has been stopped so that -a write-consistency can be guaranteed. +If you use the Admin UI, verify that the cluster health indicator is green +and that no node checks are failing. -By default write or delete operations only succeed if a quorum (> replicas / 2 -\+ 1) of active shards is available. +You can also verify the status manually. All tables should be in a green state: -:::{NOTE} -If only 1 replica is configured one active shard suffices in order for -write and delete operations to succeed. -::: +```psql +cr> SELECT * +... FROM sys.health +... WHERE health <> 'GREEN'; ++--------+----------------+-----------------+----------+------------+--------------+------------------------+ +| health | missing_shards | partition_ident | severity | table_name | table_schema | underreplicated_shards | ++--------+----------------+-----------------+----------+------------+--------------+------------------------+ ++--------+----------------+-----------------+----------+------------+--------------+------------------------+ +SELECT 0 rows in set (... sec) +``` -## Upgrade process +There should be no failed node checks: -::::::{stepper} +```psql +cr> SELECT * +... FROM sys.node_checks +... WHERE passed = FALSE; ++--------------+-------------+----+---------+--------+----------+ +| acknowledged | description | id | node_id | passed | severity | ++--------------+-------------+----+---------+--------+----------+ ++--------------+-------------+----+---------+--------+----------+ +SELECT 0 rows in set (... sec) +``` ### Backup :::{WARNING} -Before upgrading, you should {ref}`back up your data +Before upgrading, you should ensure you have {ref}`a current snapshot `. ::: +### Prevent reallocations - -### Disable allocations - -First, you have to prevent the cluster from re-distributing shards and replicas -while certain nodes are not available. You can do that by disabling -re-allocations and only allowing new primary allocations. +Prevent the cluster from unnecessarily reallocating shard replicas while nodes +are restarted. Configure the cluster to permit only the allocation of primary +shards for newly created tables or partitions. Use the {ref}`SET ` command to do so: @@ -123,72 +148,31 @@ SET OK, 1 row affected (... sec) :::{NOTE} This step may be omitted if you set the -`cluster.graceful_stop.min_availability` setting to `full`. +`cluster.graceful_stop.min_availability` setting to `full` (see {ref}`rolling_data_availability` below). ::: ### Graceful stop -To initiate a graceful shutdown that behaves as described in the introduction -of this document, the {ref}`DECOMMISSION ` -statement must be used. - -Stopping a node via the `TERM` user signal (Often invoked via `Ctrl+C` or -`systemctl stop crate`), will cause a normal shutdown of CrateDB, **without** -going through the graceful shutdown procedure described earlier. - -Depending on the size of your cluster, stopping a `crate` node gracefully -might take a while. You might want to check your server logs to see if the -graceful stop process is progressing well. In case of an error or a timeout, -the node will stay up, signaling the error in its log files (or wherever you -put your log messages). - -Using the default settings the node will shut down by moving all primary shards -off the node first. This will ensure that no data is lost. However, the cluster -health will most likely turn yellow, because replicas that lived on that node -will be missing. - -If you want to ensure green health, you need to change the -`cluster.graceful_stop.min_availability` setting to `full`. This will move -all shards off the node before shutting down. - -Keep in mind that reallocating shards might take some time depending on the -number of shards and the amount and size of records (and/or blob data). For -that reason you should set the `timeout` setting to a reasonable time. By -default the shutdown process aborts and the cluster will start distributing -shards evenly again. If you want to force a shutdown after the timeout, even if -the reallocating is not finished, you can set the `force` setting to -`true`. - -:::{WARNING} -A forced stop does not ensure the minimum data availability defined in the -settings and may result in temporary or even permanent loss of data! -::: - -:::{NOTE} -When using `cluster.graceful_stop.min_availability=full` there have to be -enough nodes in the cluster to move shards or else the graceful shutdown -procedure will fail! - -For example, if there are 4 nodes and 3 configured replicas, there will not -be enough nodes to fulfill the required replicas. +Issue an `ALTER CLUSTER DECOMMISSION` command: -Also, if there is not enough disk space on other nodes to move the shards to -the graceful stop procedure will fail. -::: +```psql +cr> ALTER CLUSTER DECOMMISSION 'your_node_name'; +ALTER OK, 1 row affected (... sec) +``` -By default, only the `graceful stop` command considers the cluster settings -described at {ref}`graceful stop `. +The `crate` process will automatically terminate at the end of the decommissioning process. -**Observe the reallocation** +:::{dropdown} **Optional: Observe the relocations** -If you want to observe the reallocation process triggered by a `full` or -`primaries` graceful-stop, you can issue the following SQL queries regularly. +If you want to observe the relocation process triggered by the graceful stop, +you can issue the following SQL queries regularly. -Get the number of shards remaining on your deallocating node: +Get the number of shards remaining on the node being decommissioned: ```psql -cr> SELECT count(*) as remaining_shards from sys.shards -... where _node['name'] = 'your_node_name'; +cr> SELECT COUNT(*) AS remaining_shards +... FROM sys.shards +... WHERE node['name'] = 'your_node_name'; +------------------+ | remaining_shards | +------------------+ @@ -197,56 +181,41 @@ cr> SELECT count(*) as remaining_shards from sys.shards SELECT 1 row in set (... sec) ``` -Get some more details about what shards are remaining on your node: +Get some more details about which shards remain on your node: ```psql -cr> SELECT schema_name as schema, table_name as "table", id, "primary", state +cr> SELECT schema_name, table_name, partition_ident, id, primary, size / POWER(1024, 3) AS size_gb, state ... FROM sys.shards -... WHERE _node['name'] = 'your_node_name' AND schema_name IN ('blob', 'doc') -... ORDER BY schema, "table", id, "primary", state; -+--------+-------+----+---------+-------+ -| schema | table | id | primary | state | -+--------+-------+----+---------+-------+ +... WHERE node['name'] = 'your_node_name' +... ORDER BY 1, 2, 3, 4, 5; ++-------------+------------+-----------------+----+---------+---------+-------+ +| schema_name | table_name | partition_ident | id | primary | size_gb | state | ++-------------+------------+-----------------+----+---------+---------+-------+ ... SELECT ... rows in set (... sec) ``` -In the case of `primaries` availability, only the primary shards of tables -with zero replicas will be reallocated. Use this query to find out which shards -to look for: - -```psql -cr> SELECT table_schema as schema, table_name as "table" -... FROM information_schema.tables -... WHERE number_of_replicas = 0 and table_schema in ('blob', 'doc') -... ORDER BY schema, "table" ; -+--------+-------...+ -| schema | table ...| -+--------+-------...+ -... -+--------+-------...+ -SELECT ... rows in set (... sec) -``` +::: :::{NOTE} -If you observe the graceful-stop process using the admin UI, you might see +If you observe the graceful stop process using the Admin UI, you might see the cluster turning red for a small instant when a node finally shuts down. -This is due to the way the admin UI determines the cluster state. +This is due to the way the Admin UI determines the cluster state. -If a query fails due to a missing node, the admin UI may falsely consider +If a query fails due to a missing node, the Admin UI may falsely consider the cluster to be in a critical state. ::: ### Upgrade CrateDB -After the node is stopped you can safely upgrade your CrateDB installation. -Depending on your installation and operating system you can do it by -downloading the latest tarball or just use the package manager. +After the node has stopped, you can safely upgrade your CrateDB installation. +Depending on your installation and operating system, you can upgrade using the +package manager. -Example for RHEL/YUM: +Example for RHEL (DNF package manager): -``` -$sh yum update -y crate +```shell +dnf update -y crate ``` If you are in doubt how to upgrade an installed package, please refer to the @@ -254,33 +223,116 @@ man pages of your operating system or package manager. ### Start CrateDB -Once the upgrade process is completed you can start the CrateDB process again -by either invoking the bin/crate executable from the tarball directly: +Once the upgrade has completed, you can start the CrateDB process again. +Most commonly, this is done using your operating system's service manager, such as: -``` -sh$ /path/to/bin/crate +```shell +systemctl start crate ``` -Or using the service manager of your operating system. +### Wait for the node to rejoin -Example for RHEL/YUM: +Wait until the upgraded node has started and rejoined the cluster. +Once the node rejoined, it will appear in `sys.nodes` with the updated version number: + +```psql +cr> SELECT version['number'] +... FROM sys.nodes +... WHERE name = 'your_node_name'; ++-------------------+ +| version['number'] | ++-------------------+ +... +SELECT ... rows in set (... sec) ``` -sh$ service crate start -``` + +Confirm that the query returns exactly one row containing the target version. ### Repeat -Repeat steps 2, 3, and 4 for all other nodes. +Repeat steps 4 - 7 for all other nodes. ### Enable allocations -Finally, when all nodes are updated you can re-enable allocations -again that have been disabled in the first step: +Finally, when all nodes are updated you can restore the shard allocation +setting. If allocations were set to the default value before, run this query +to restore the setting: ```psql cr> SET GLOBAL TRANSIENT "cluster.routing.allocation.enable" = 'all'; SET OK, 1 row affected (... sec) ``` +### Wait for complete recovery + +Before considering the upgrade done, verify that all shards have successfully +been recovered. This process can take some time, depending on your data volume. +The number of tables or partitions with non-green health should steadily +decrease as shards recover. + +```psql +cr> SELECT * +... FROM sys.health +... WHERE health <> 'GREEN'; ++--------+----------------+-----------------+----------+------------+--------------+------------------------+ +| health | missing_shards | partition_ident | severity | table_name | table_schema | underreplicated_shards | ++--------+----------------+-----------------+----------+------------+--------------+------------------------+ +... +SELECT 0 rows in set (... sec) +``` + :::::: + +(rolling_data_availability)= + +## Data availability options + +There are different levels of data availability that can be achieved during the upgrade process. +This is controlled via the {ref}`cluster.graceful_stop.min_availability ` +parameter. It can be either `primaries`, `full`, or `none` and can be configured using the +{ref}`SET ` statement. + +- **primaries** *(default)*: Only primary shards will be moved to other nodes. Using +this setting means that the cluster will go into the `yellow` (underreplicated) warning state +if a node that has been stopped contained replicas that are then unavailable. +- **full**: All shards currently located on the node will be moved to the +other nodes in order to stop gracefully. Using this setting, the cluster will +stay `green` the whole time. +- **none**: There is no data-availability guarantee. The node will stop, +possibly leaving the cluster in the critical `red` state if the node +contained a primary shard that has no replicas that can take over. + +The default `primaries` setting offers a good balance between relocating shards and data availability. +Choose `none` if you prefer a fast upgrade path and can tolerate intermittent data unavailability +(e.g. during a maintenance window), or `full` if you want to retain full availability at all times. + +### Full minimum data availability + +If the `full` minimum data availability is configured, the cluster needs to +contain enough nodes to hold the number of replicas that are configured, even if +one node is missing. + +For example, if there are only two nodes in a cluster and a table has one +replica configured, the graceful stop procedure will not succeed and abort +as it won't be possible to relocate the replicas. + +If a table has a range configured as number of replicas, the upper number of +replicas will be taken into account. +With two nodes and 0-1 replicas, the graceful stop procedure will abort. + +:::{NOTE} +For the `full` graceful stop to work, the following has to be true: + +**number_of_nodes > max_number_of_replicas + 1** +::: + +### Primaries minimum data availability + +If the `primaries` minimum data availability is used, take care that there +are still enough replicas in the cluster after a node has been stopped so that +writes can be processed. + +:::{NOTE} +By default, write or delete operations succeed if the primary shard is available (see `CREATE TABLE` parameter {ref}`write.wait-for-active-shards `). +::: From e08670af18310e4b8cfc6cfa4ea792aae63f0314 Mon Sep 17 00:00:00 2001 From: Niklas Schmidtmer Date: Fri, 17 Jul 2026 13:56:38 +0200 Subject: [PATCH 2/3] Change allocation recommendation to primaries instead of new_primaries --- docs/admin/upgrade/rolling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/upgrade/rolling.md b/docs/admin/upgrade/rolling.md index 9036c324..6c3bce5d 100644 --- a/docs/admin/upgrade/rolling.md +++ b/docs/admin/upgrade/rolling.md @@ -142,7 +142,7 @@ shards for newly created tables or partitions. Use the {ref}`SET ` command to do so: ```psql -cr> SET GLOBAL TRANSIENT "cluster.routing.allocation.enable" = 'new_primaries'; +cr> SET GLOBAL TRANSIENT "cluster.routing.allocation.enable" = 'primaries'; SET OK, 1 row affected (... sec) ``` From 08f39ec436e65615ef849423213f03f90586193a Mon Sep 17 00:00:00 2001 From: Niklas Schmidtmer Date: Tue, 21 Jul 2026 16:39:19 +0200 Subject: [PATCH 3/3] Simplify operating system reference --- docs/admin/upgrade/rolling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/admin/upgrade/rolling.md b/docs/admin/upgrade/rolling.md index 6c3bce5d..b98cfc4a 100644 --- a/docs/admin/upgrade/rolling.md +++ b/docs/admin/upgrade/rolling.md @@ -218,8 +218,8 @@ Example for RHEL (DNF package manager): dnf update -y crate ``` -If you are in doubt how to upgrade an installed package, please refer to the -man pages of your operating system or package manager. +If you are in doubt how to upgrade an installed package, please refer to your +operating system documentation. ### Start CrateDB