From ef5bfc5fa8681a074d20ac01991d455e794ad5c4 Mon Sep 17 00:00:00 2001 From: yahavx-ms <80623486+yahavx-ms@users.noreply.github.com> Date: Mon, 18 Nov 2024 10:41:27 +0200 Subject: [PATCH 01/10] Update capacity-policy.md --- data-explorer/kusto/management/capacity-policy.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/capacity-policy.md b/data-explorer/kusto/management/capacity-policy.md index 4164c5e07e..ee4ffaccd0 100644 --- a/data-explorer/kusto/management/capacity-policy.md +++ b/data-explorer/kusto/management/capacity-policy.md @@ -51,18 +51,22 @@ The [.show capacity](show-capacity-command.md) command returns the cluster's ing |--|--|--| | `MinimumConcurrentOperationsPerNode` | `long` | The minimal number of concurrent extents merge/rebuild operations on a single node. Default is `1`. | | `MaximumConcurrentOperationsPerNode` |`long` | The maximum number of concurrent extents merge/rebuild operations on a single node. Default is `5`. | +| `ClusterMaximumConcurrentOperations` | `long` | The maximum number of concurrent extents merge/rebuild operations allowed in a cluster. This value caps the total merge capacity, as shown in the following formula. | **Formula** The [.show capacity](show-capacity-command.md) command returns the cluster's extents merge capacity based on the following formula: -*Number of nodes in cluster* `*` *Concurrent operations per node* +`Minimum([Number of nodes in cluster * Concurrent operations per node], ClusterMaximumConcurrentOperations)` The effective value for *Concurrent operations per node* is automatically adjusted by the system in the range [`MinimumConcurrentOperationsPerNode`,`MaximumConcurrentOperationsPerNode`], as long as the success rate of the merge operations is 90% or higher. > [!NOTE] > In clusters with four or more nodes, the admin node doesn't participate in merge operations, so *Number of nodes in cluster* is reduced by one. +> [!WARNING] +> Consult with the support team before altering the extents merge capacity policy. + ### Extents purge rebuild capacity | Property | Type | Description | From 6c41098798013addcdf1aaa6bac43a40251d8263 Mon Sep 17 00:00:00 2001 From: yahavx-ms <80623486+yahavx-ms@users.noreply.github.com> Date: Mon, 10 Feb 2025 14:06:47 +0200 Subject: [PATCH 02/10] Update data-explorer/kusto/management/capacity-policy.md Co-authored-by: Meira Josephy <144697924+mjosephym@users.noreply.github.com> --- data-explorer/kusto/management/capacity-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/capacity-policy.md b/data-explorer/kusto/management/capacity-policy.md index ee4ffaccd0..aff5fe14f6 100644 --- a/data-explorer/kusto/management/capacity-policy.md +++ b/data-explorer/kusto/management/capacity-policy.md @@ -51,7 +51,7 @@ The [.show capacity](show-capacity-command.md) command returns the cluster's ing |--|--|--| | `MinimumConcurrentOperationsPerNode` | `long` | The minimal number of concurrent extents merge/rebuild operations on a single node. Default is `1`. | | `MaximumConcurrentOperationsPerNode` |`long` | The maximum number of concurrent extents merge/rebuild operations on a single node. Default is `5`. | -| `ClusterMaximumConcurrentOperations` | `long` | The maximum number of concurrent extents merge/rebuild operations allowed in a cluster. This value caps the total merge capacity, as shown in the following formula. | +| `ClusterMaximumConcurrentOperations` | `long` | The maximum number of concurrent extents merge/rebuild operations allowed in a cluster. This value caps the total merge capacity. See the following formula for an example. | **Formula** From df5cfaaec438dad48f32507498fb354a9d9c608f Mon Sep 17 00:00:00 2001 From: yahavx-ms <80623486+yahavx-ms@users.noreply.github.com> Date: Mon, 10 Feb 2025 14:06:54 +0200 Subject: [PATCH 03/10] Update data-explorer/kusto/management/capacity-policy.md Co-authored-by: Meira Josephy <144697924+mjosephym@users.noreply.github.com> --- data-explorer/kusto/management/capacity-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/capacity-policy.md b/data-explorer/kusto/management/capacity-policy.md index aff5fe14f6..7877158650 100644 --- a/data-explorer/kusto/management/capacity-policy.md +++ b/data-explorer/kusto/management/capacity-policy.md @@ -57,7 +57,7 @@ The [.show capacity](show-capacity-command.md) command returns the cluster's ing The [.show capacity](show-capacity-command.md) command returns the cluster's extents merge capacity based on the following formula: -`Minimum([Number of nodes in cluster * Concurrent operations per node], ClusterMaximumConcurrentOperations)` +`Minimum(`*Number of nodes in cluster* `*` *Concurrent operations per node*`,` `ClusterMaximumConcurrentOperations)` The effective value for *Concurrent operations per node* is automatically adjusted by the system in the range [`MinimumConcurrentOperationsPerNode`,`MaximumConcurrentOperationsPerNode`], as long as the success rate of the merge operations is 90% or higher. From 04ffaf0b59ba6cfa1116a23044fb381e99fbb623 Mon Sep 17 00:00:00 2001 From: yahavx-ms <80623486+yahavx-ms@users.noreply.github.com> Date: Mon, 10 Feb 2025 14:11:05 +0200 Subject: [PATCH 04/10] Update description for ClusterMaximumConcurrentOperations --- data-explorer/kusto/management/capacity-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/capacity-policy.md b/data-explorer/kusto/management/capacity-policy.md index 7877158650..f50c094f6b 100644 --- a/data-explorer/kusto/management/capacity-policy.md +++ b/data-explorer/kusto/management/capacity-policy.md @@ -51,7 +51,7 @@ The [.show capacity](show-capacity-command.md) command returns the cluster's ing |--|--|--| | `MinimumConcurrentOperationsPerNode` | `long` | The minimal number of concurrent extents merge/rebuild operations on a single node. Default is `1`. | | `MaximumConcurrentOperationsPerNode` |`long` | The maximum number of concurrent extents merge/rebuild operations on a single node. Default is `5`. | -| `ClusterMaximumConcurrentOperations` | `long` | The maximum number of concurrent extents merge/rebuild operations allowed in a cluster. This value caps the total merge capacity. See the following formula for an example. | +| `ClusterMaximumConcurrentOperations` | `long` | The maximum number of concurrent extents merge/rebuild operations allowed in a cluster. This value caps the total merge capacity. See the below formula for an example. | **Formula** From 53ed06870e7a0488444620cc05fb30535c014e42 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Tue, 11 Feb 2025 11:00:11 +0200 Subject: [PATCH 05/10] capacity updates --- .../kusto/management/capacity-policy.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/data-explorer/kusto/management/capacity-policy.md b/data-explorer/kusto/management/capacity-policy.md index 16e979ce3e..4f0ad2be77 100644 --- a/data-explorer/kusto/management/capacity-policy.md +++ b/data-explorer/kusto/management/capacity-policy.md @@ -3,7 +3,7 @@ title: Capacity policy description: Learn how to use the capacity policy to control the compute resources of data management operations on a cluster. ms.reviewer: alexans ms.topic: reference -ms.date: 01/06/2025 +ms.date: 02/11/2025 monikerRange: "azure-data-explorer" --- # Capacity policy @@ -51,7 +51,7 @@ The [.show capacity](show-capacity-command.md) command returns the cluster's ing |--|--|--| | `MinimumConcurrentOperationsPerNode` | `long` | The minimal number of concurrent extents merge/rebuild operations on a single node. Default is `1`. | | `MaximumConcurrentOperationsPerNode` |`long` | The maximum number of concurrent extents merge/rebuild operations on a single node. Default is `5`. | -| `ClusterMaximumConcurrentOperations` | `long` | The maximum number of concurrent extents merge/rebuild operations allowed in a cluster. This value caps the total merge capacity. See the below formula for an example. | +| `ClusterMaximumConcurrentOperations` | `long` | The maximum number of concurrent extents merge/rebuild operations allowed in a cluster. This value caps the total merge capacity. For an example, see the following formula. | **Formula** @@ -118,7 +118,7 @@ The policy can be used to change concurrency settings for [materialized views](m | `ClusterMinimumConcurrentOperations` | `long` | The minimal number of concurrent materialization operations in a cluster. Default is `1`. | | `ClusterMaximumConcurrentOperations` | `long` | The maximum number of concurrent materialization operations in a cluster. Default is `10`. | -By default, only a single materialization runs concurrently (see [how materialized views work](materialized-views/materialized-view-overview.md#how-materialized-views-work)). The system adjusts the current concurrency in the range [`ClusterMinimumConcurrentOperations`,`ClusterMaximumConcurrentOperations`], based on the number of materialized views in the cluster and the cluster's CPU. You can increase/decrease concurrency by altering this policy. For example, if the cluster has ten materialized views, setting the `ClusterMinimumConcurrentOperations` to five ensures that at least five of them can materialize concurrently. +By default, only a single materialization runs concurrently (see [how materialized views work](materialized-views/materialized-view-overview.md#how-materialized-views-work)). The system adjusts the current concurrency in the range [`ClusterMinimumConcurrentOperations`,`ClusterMaximumConcurrentOperations`], based on the number of materialized views in the cluster and the cluster's CPU. You can increase/decrease concurrency by altering this policy. For example, if the cluster has 10 materialized views, setting the `ClusterMinimumConcurrentOperations` to five ensures that at least five of them can materialize concurrently. You can view the effective value for the current concurrency using the [.show capacity command](show-capacity-command.md) > [!WARNING] @@ -230,7 +230,7 @@ The default capacity policy has the following JSON representation: ## Management commands > [!WARNING] -> Consult with the support team before altering a capacity policy. +> Before altering the extents policy, we recommend that you [consult with support](https://ms.portal.azure.com/#create/Microsoft.Support). * Use [`.show cluster policy capacity`](show-cluster-capacity-policy-command.md) to show the current capacity policy of the cluster. * Use [`.alter-merge cluster policy capacity`](alter-merge-capacity-policy-command.md) to alter the capacity policy of the cluster. @@ -241,23 +241,23 @@ Kusto limits the number of concurrent requests for the following user-initiated * **Ingestions** * This category includes commands that [ingest from storage](data-ingestion/ingest-into-command.md), [ingest from a query](data-ingestion/ingest-from-query.md), and [ingest inline](data-ingestion/ingest-inline.md). - * The limit is as defined by the [ingestion capacity](#ingestion-capacity). + * The limit is defined by the [ingestion capacity](#ingestion-capacity). * **Purges** * The global limit is currently fixed at one per cluster. - * The [purge rebuild capacity](#extents-purge-rebuild-capacity) is used internally to determine the number of concurrent rebuild operations during purge commands. Purge commands won't be blocked or throttled because of this process, but will complete faster or slower depending on the purge rebuild capacity. + * The [purge rebuild capacity](#extents-purge-rebuild-capacity) is used internally to determine the number of concurrent rebuild operations during purge commands. Purge commands aren't blocked or throttled because of this process, but completes faster or slower depending on the purge rebuild capacity. * **Exports** * The limit is as defined in the [export capacity](#export-capacity). -When the cluster detects that an operation has exceeded the limit on concurrent requests: +When the cluster detects that an operation exceeded the limit on concurrent requests: * The command's state, as presented by [System information commands](system-info.md), is `Throttled`. -* The error message includes the *command type*, the *origin* of the throttling and the *capacity* that's been exceeded. For example: +* The error message includes the *command type*, the *origin* of the throttling and the *capacity* that exceeded. For example: * For example: `The management command was aborted due to throttling. Retrying after some backoff might succeed. CommandType: 'TableSetOrAppend', Capacity: 18, Origin: 'CapacityPolicy/Ingestion'`. * The HTTP response code is `429`. The subcode is `TooManyRequests`. * The exception type is `ControlCommandThrottledException`. > [!NOTE] -> Management commands may also be throttled as a result of exceeding the limit defined by a workload group's [Request rate limit policy](request-rate-limit-policy.md). +> Management commands can also be throttled as a result of exceeding the limit defined by a workload group's [Request rate limit policy](request-rate-limit-policy.md). ## Related content From 39416922c3673a489efcc158b0314fa14085a3cc Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Tue, 11 Feb 2025 11:57:56 +0200 Subject: [PATCH 06/10] edits --- data-explorer/kusto/management/capacity-policy.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/data-explorer/kusto/management/capacity-policy.md b/data-explorer/kusto/management/capacity-policy.md index 4f0ad2be77..26a19c10dd 100644 --- a/data-explorer/kusto/management/capacity-policy.md +++ b/data-explorer/kusto/management/capacity-policy.md @@ -65,7 +65,7 @@ The effective value for *Concurrent operations per node* is automatically adjust > In clusters with four or more nodes, the admin node doesn't participate in merge operations, so *Number of nodes in cluster* is reduced by one. > [!WARNING] -> Consult with the support team before altering the extents merge capacity policy. +> Before altering the extents merge policy, we recommend that you [consult with support](https://ms.portal.azure.com/#create/Microsoft.Support). ### Extents purge rebuild capacity @@ -241,7 +241,7 @@ Kusto limits the number of concurrent requests for the following user-initiated * **Ingestions** * This category includes commands that [ingest from storage](data-ingestion/ingest-into-command.md), [ingest from a query](data-ingestion/ingest-from-query.md), and [ingest inline](data-ingestion/ingest-inline.md). - * The limit is defined by the [ingestion capacity](#ingestion-capacity). + * The [ingestion capacity](#ingestion-capacity) defines the limit. * **Purges** * The global limit is currently fixed at one per cluster. * The [purge rebuild capacity](#extents-purge-rebuild-capacity) is used internally to determine the number of concurrent rebuild operations during purge commands. Purge commands aren't blocked or throttled because of this process, but completes faster or slower depending on the purge rebuild capacity. @@ -257,7 +257,7 @@ When the cluster detects that an operation exceeded the limit on concurrent requ * The exception type is `ControlCommandThrottledException`. > [!NOTE] -> Management commands can also be throttled as a result of exceeding the limit defined by a workload group's [Request rate limit policy](request-rate-limit-policy.md). +> Management commands can also be throttled if they exceed the limits set by a workload group's [Request rate limit policy](request-rate-limit-policy.md). ## Related content From 6ede0bece43500566d992a9969e4fa4fbcf4f54d Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Tue, 11 Feb 2025 12:09:38 +0200 Subject: [PATCH 07/10] edits --- data-explorer/kusto/management/capacity-policy.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/management/capacity-policy.md b/data-explorer/kusto/management/capacity-policy.md index 26a19c10dd..572bd35cd2 100644 --- a/data-explorer/kusto/management/capacity-policy.md +++ b/data-explorer/kusto/management/capacity-policy.md @@ -65,7 +65,7 @@ The effective value for *Concurrent operations per node* is automatically adjust > In clusters with four or more nodes, the admin node doesn't participate in merge operations, so *Number of nodes in cluster* is reduced by one. > [!WARNING] -> Before altering the extents merge policy, we recommend that you [consult with support](https://ms.portal.azure.com/#create/Microsoft.Support). +> Given the complexity of the extents merge policy, we recommend that you [consult with support](https://ms.portal.azure.com/#create/Microsoft.Support) before making any changes. ### Extents purge rebuild capacity @@ -230,7 +230,7 @@ The default capacity policy has the following JSON representation: ## Management commands > [!WARNING] -> Before altering the extents policy, we recommend that you [consult with support](https://ms.portal.azure.com/#create/Microsoft.Support). +> Given the complexity of the capacity policy, we recommend that you [consult with support](https://ms.portal.azure.com/#create/Microsoft.Support) before making any changes. * Use [`.show cluster policy capacity`](show-cluster-capacity-policy-command.md) to show the current capacity policy of the cluster. * Use [`.alter-merge cluster policy capacity`](alter-merge-capacity-policy-command.md) to alter the capacity policy of the cluster. From 21c1386569812e170fa60dc0aff27a0a37bf402c Mon Sep 17 00:00:00 2001 From: Yifat Schachter Date: Wed, 12 Feb 2025 13:31:03 +0200 Subject: [PATCH 08/10] update policy docs fix --- data-explorer/kusto/management/update-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/update-policy.md b/data-explorer/kusto/management/update-policy.md index 7898a092d7..fcb112f158 100644 --- a/data-explorer/kusto/management/update-policy.md +++ b/data-explorer/kusto/management/update-policy.md @@ -27,7 +27,7 @@ An update policy is subject to the same restrictions and best practices as regul > [!NOTE] > > * The source and target table must be in the same database. -> * The update policy function schema and the target table schema must match in their column names, types, and order. +> * The update policy function schema and the target table schema must match in their column types, and order. > * The update policy function can reference tables in other databases. To do this, the update policy must be defined with a `ManagedIdentity` property, and the managed identity must have `viewer` [role](security-roles.md) on the referenced databases. Ingesting formatted data improves performance, and CSV is preferred because of it's a well-defined format. Sometimes, however, you have no control over the format of the data, or you want to enrich ingested data, for example, by joining records with a static dimension table in your database. From ca5b13ddf728d564d8cd9240df05a70d9f3ca45d Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Sun, 16 Feb 2025 12:32:23 +0200 Subject: [PATCH 09/10] Update data-explorer/kusto/management/capacity-policy.md --- data-explorer/kusto/management/capacity-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/capacity-policy.md b/data-explorer/kusto/management/capacity-policy.md index 572bd35cd2..cc40ab4642 100644 --- a/data-explorer/kusto/management/capacity-policy.md +++ b/data-explorer/kusto/management/capacity-policy.md @@ -65,7 +65,7 @@ The effective value for *Concurrent operations per node* is automatically adjust > In clusters with four or more nodes, the admin node doesn't participate in merge operations, so *Number of nodes in cluster* is reduced by one. > [!WARNING] -> Given the complexity of the extents merge policy, we recommend that you [consult with support](https://ms.portal.azure.com/#create/Microsoft.Support) before making any changes. +> Due to the complexity of the extents merge policy, we recommend that you [consult with support](https://ms.portal.azure.com/#create/Microsoft.Support) before making any changes. ### Extents purge rebuild capacity From 56ef534ce205f11fb33025a61ca625a359492f35 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Sun, 16 Feb 2025 12:49:01 +0200 Subject: [PATCH 10/10] Update data-explorer/kusto/management/capacity-policy.md --- data-explorer/kusto/management/capacity-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/capacity-policy.md b/data-explorer/kusto/management/capacity-policy.md index cc40ab4642..508d926c28 100644 --- a/data-explorer/kusto/management/capacity-policy.md +++ b/data-explorer/kusto/management/capacity-policy.md @@ -51,7 +51,7 @@ The [.show capacity](show-capacity-command.md) command returns the cluster's ing |--|--|--| | `MinimumConcurrentOperationsPerNode` | `long` | The minimal number of concurrent extents merge/rebuild operations on a single node. Default is `1`. | | `MaximumConcurrentOperationsPerNode` |`long` | The maximum number of concurrent extents merge/rebuild operations on a single node. Default is `5`. | -| `ClusterMaximumConcurrentOperations` | `long` | The maximum number of concurrent extents merge/rebuild operations allowed in a cluster. This value caps the total merge capacity. For an example, see the following formula. | +| `ClusterMaximumConcurrentOperations` | `long` | The maximum number of concurrent extents merge/rebuild operations allowed in a cluster. This value caps the total merge capacity. | **Formula**