From 76f7782af35f0a5b7b5b29baa0271f8e0ff365ce Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Sun, 26 Jan 2025 08:53:55 +0200 Subject: [PATCH 01/27] Learn Editor: Update external-tables-azure-storage.md --- .../kusto/management/external-tables-azure-storage.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/external-tables-azure-storage.md b/data-explorer/kusto/management/external-tables-azure-storage.md index daaaed866c..659eb7e003 100644 --- a/data-explorer/kusto/management/external-tables-azure-storage.md +++ b/data-explorer/kusto/management/external-tables-azure-storage.md @@ -122,7 +122,8 @@ external_table("ExternalTable") |------------------|----------|-------------------------------------------------------------------------------------| | `folder` | `string` | Table's folder | | `docString` | `string` | String documenting the table | -| `compressed` | `bool` | If set, indicates whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only) | +| `compressed` | `bool` | If set, indicates whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression will be auto detected based on the file) | +| `compressionType` | `string` | If set, indicates the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression will be auto detected based on the file) | | `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), indicates whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). | | `namePrefix` | `string` | If set, indicates the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | | `fileExtension` | `string` | If set, indicates file extensions of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | From f95d040eb6c9d903b03c31d01e7c2ef21c9db7cf Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Sun, 26 Jan 2025 08:58:36 +0200 Subject: [PATCH 02/27] Learn Editor: Update show-external-tables.md --- data-explorer/kusto/management/show-external-tables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/show-external-tables.md b/data-explorer/kusto/management/show-external-tables.md index 40bc28b1e5..6149e0fa06 100644 --- a/data-explorer/kusto/management/show-external-tables.md +++ b/data-explorer/kusto/management/show-external-tables.md @@ -39,7 +39,7 @@ You must have at least Database User, Database Viewer, Database Monitor to run t | TableType | `string` | Type of external table | | Folder | `string` | Table's folder | | DocString | `string` | String documenting the table | -| Properties | `string` | Table's JSON serialized properties (specific to the type of table) | +| Properties | `string` | Table's JSON serialized properties (specific to the type of table; visit "Create or alter external table" for more information) | ## Example From 40bb7454676b6107079128ea575f1a5ea42dd9f5 Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Sun, 26 Jan 2025 09:06:28 +0200 Subject: [PATCH 03/27] Learn Editor: Update export-data-to-storage.md --- .../kusto/management/data-export/export-data-to-storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md index 775c0238f9..6694732082 100644 --- a/data-explorer/kusto/management/data-export/export-data-to-storage.md +++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md @@ -42,7 +42,7 @@ You must have at least [Database Viewer](../../access-control/role-based-access- | `fileExtension` | `string` | Indicates the "extension" part of the storage artifact (for example, `.csv` or `.tsv`). If compression is used, `.gz` is appended as well. | | `namePrefix` | `string` | Indicates a prefix to add to each generated storage artifact name. A random prefix is used if left unspecified. | | `encoding` | `string` | Indicates how to encode the text: `UTF8NoBOM` (default) or `UTF8BOM`. | -| `compressionType` | `string` | Indicates the type of compression to use. Possible values are `gzip` or `snappy`. Default is `gzip`. `snappy` can (optionally) be used for `parquet` format. | +| `compressionType` | `string` | Indicates the type of compression to use. Possible values are `gzip`, and for `parquet` format `snappy`, `lz4_raw`, `brotli`, or `zstd` as well. Default is `gzip`. | | `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. | | `persistDetails` | `bool` | Indicates that the command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | | `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. | From 7aa9fc2aed76c29b58554de1c24840a450cea52a Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Sun, 26 Jan 2025 13:08:49 +0200 Subject: [PATCH 04/27] Learn Editor: Update export-data-to-storage.md --- .../kusto/management/data-export/export-data-to-storage.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md index 6694732082..83e7227a59 100644 --- a/data-explorer/kusto/management/data-export/export-data-to-storage.md +++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md @@ -25,8 +25,8 @@ You must have at least [Database Viewer](../../access-control/role-based-access- | Name | Type | Required | Description | |--|--|--|--| -| `async` | `string` | | If specified, the command runs in asynchronous mode. See [asynchronous mode](#asynchronous-mode). | -| `compressed` | `string` | | If specified, the output storage artifacts are compressed as `.gz` files. See the `compressionType` [supported property](#supported-properties) for compressing Parquet files as snappy. | +| *async* | `string` | | If specified, the command runs in asynchronous mode. See [asynchronous mode](#asynchronous-mode). | +| *compressed* | `bool` | | If specified, the output storage artifacts are compressed as `.gz` files. See the `compressionType` [supported property](#supported-properties) for compressing Parquet files as snappy. | | *OutputDataFormat* | `string` | :heavy_check_mark: | Indicates the data format of the storage artifacts written by the command. Supported values are: `csv`, `tsv`, `json`, and `parquet`. | | *StorageConnectionString* | `string` | | One or more [storage connection strings](../../api/connection-strings/storage-connection-strings.md) that indicate which storage to write the data to. More than one storage connection string might be specified for scalable writes. Each such connection string must indicate the credentials to use when writing to storage. For example, when writing to Azure Blob Storage, the credentials can be the storage account key, or a shared access key (SAS) with the permissions to read, write, and list blobs. | | *PropertyName*, *PropertyValue* | `string` | | A comma-separated list of key-value property pairs. See [supported properties](#supported-properties).| From 723e43e10608bd108f375254986f1313269c9a98 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Sun, 26 Jan 2025 18:11:39 +0200 Subject: [PATCH 05/27] replace branch PR 6542 --- data-explorer/kusto/query/project-operator.md | 10 ++++++++-- data-explorer/kusto/query/project-rename-operator.md | 4 +++- data-explorer/kusto/query/project-reorder-operator.md | 6 ++++++ .../kusto/query/query-parameters-statement.md | 2 +- data-explorer/kusto/query/restrict-statement.md | 10 +++++----- data-explorer/kusto/query/set-statement.md | 2 +- .../kusto/query/tabular-expression-statements.md | 2 +- 7 files changed, 25 insertions(+), 11 deletions(-) diff --git a/data-explorer/kusto/query/project-operator.md b/data-explorer/kusto/query/project-operator.md index 80588280ca..187d5caca1 100644 --- a/data-explorer/kusto/query/project-operator.md +++ b/data-explorer/kusto/query/project-operator.md @@ -47,6 +47,8 @@ A table with columns that were named as arguments. Contains same number of rows ## Examples +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + ### Only show specific columns Only show the `EventId`, `State`, `EventType` of the `StormEvents` table. @@ -61,7 +63,9 @@ StormEvents | project EventId, State, EventType ``` -The following results table shows only the top 10 results. +**Output** + +The table shows only the top 10 results. |EventId|State|EventType| |--|--|--| @@ -92,7 +96,9 @@ StormEvents | where TotalInjuries > 5 ``` -The following table shows only the first 10 results. +**Output** + +The table shows only the first 10 results. |StartLocation| TotalInjuries| |--|--| diff --git a/data-explorer/kusto/query/project-rename-operator.md b/data-explorer/kusto/query/project-rename-operator.md index 87f59c0353..5baf3ec62b 100644 --- a/data-explorer/kusto/query/project-rename-operator.md +++ b/data-explorer/kusto/query/project-rename-operator.md @@ -3,7 +3,7 @@ title: project-rename operator description: Learn how to use the project-rename operator to rename columns in the output table. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/20/2025 --- # project-rename operator @@ -31,6 +31,8 @@ A table that has the columns in the same order as in an existing table, with col ## Examples +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] > Run the query diff --git a/data-explorer/kusto/query/project-reorder-operator.md b/data-explorer/kusto/query/project-reorder-operator.md index a20a781de8..863c2efc93 100644 --- a/data-explorer/kusto/query/project-reorder-operator.md +++ b/data-explorer/kusto/query/project-reorder-operator.md @@ -40,6 +40,10 @@ A table that contains columns in the order specified by the operator arguments. ## Examples +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] + +### Reorder with b first + Reorder a table with three columns (a, b, c) so the second column (b) will appear first. :::moniker range="azure-data-explorer" @@ -58,6 +62,8 @@ print a='a', b='b', c='c' |---|---|---| |b|a|c| +### Reorder with a first + Reorder columns of a table so that columns starting with `a` will appear before other columns. :::moniker range="azure-data-explorer" diff --git a/data-explorer/kusto/query/query-parameters-statement.md b/data-explorer/kusto/query/query-parameters-statement.md index 3eb7e68154..694f38fbc3 100644 --- a/data-explorer/kusto/query/query-parameters-statement.md +++ b/data-explorer/kusto/query/query-parameters-statement.md @@ -44,7 +44,7 @@ To reference query parameters, the query text, or functions it uses, must first ## Example -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] diff --git a/data-explorer/kusto/query/restrict-statement.md b/data-explorer/kusto/query/restrict-statement.md index e7db0058ed..d0ac82e878 100644 --- a/data-explorer/kusto/query/restrict-statement.md +++ b/data-explorer/kusto/query/restrict-statement.md @@ -3,7 +3,7 @@ title: Restrict statement description: Learn how to use the restrict statement to limit tabular views that are visible to subsequent query statements. ms.reviewer: alexans ms.topic: reference -ms.date: 01/13/2025 +ms.date: 01/20/2025 monikerRange: "microsoft-fabric || azure-data-explorer" --- # Restrict statement @@ -14,10 +14,10 @@ The restrict statement limits the set of table/view entities which are visible t The restrict statement's main scenario is for middle-tier applications that accept queries from users and want to apply a row-level security mechanism over those queries. -The middle-tier application can prefix the user's query with a **logical model**, a set of let statements defining views that restrict the user's access to data, for example ( `T | where UserId == "..."`). As the last statement being added, it restricts the user's access to the logical model only. +The middle-tier application can prefix the user's query with a **logical model**, a set of let statements to define views that restrict the user's access to data, for example ( `T | where UserId == "..."`). As the last statement being added, it restricts the user's access to the logical model only. > [!NOTE] -> The restrict statement can be used to restrict access to entities in another database or cluster (wildcards are not supported in cluster names). +> The restrict statement can be used to restrict access to entities in another database or cluster (wildcards aren't supported in cluster names). ## Syntax @@ -34,11 +34,11 @@ The middle-tier application can prefix the user's query with a **logical model** > [!NOTE] > > * All tables, tabular views, or patterns that aren't specified by the restrict statement become "invisible" to the rest of the query. -> * Let, set, and tabular statements are strung together/separated by a semicolon, otherwise they won't be considered part of the same query. +> * Let, set, and tabular statements are strung together/separated by a semicolon, otherwise they aren't considered part of the same query. ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Let statement diff --git a/data-explorer/kusto/query/set-statement.md b/data-explorer/kusto/query/set-statement.md index 4aceeb710a..49d2095c28 100644 --- a/data-explorer/kusto/query/set-statement.md +++ b/data-explorer/kusto/query/set-statement.md @@ -35,7 +35,7 @@ Request properties aren't formally a part of the Kusto Query Language and may be ## Example -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ```kusto set querytrace; diff --git a/data-explorer/kusto/query/tabular-expression-statements.md b/data-explorer/kusto/query/tabular-expression-statements.md index 3b5d738cb5..4afdee8191 100644 --- a/data-explorer/kusto/query/tabular-expression-statements.md +++ b/data-explorer/kusto/query/tabular-expression-statements.md @@ -41,7 +41,7 @@ A tabular data source produces sets of records, to be further processed by tabul ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Filter rows by condition From 6f0489e96f79fd935041ebd806e01d81da40c9b6 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 28 Jan 2025 16:32:08 +0200 Subject: [PATCH 06/27] Replace help cluster include --- data-explorer/kusto/query/let-statement.md | 4 ++-- data-explorer/kusto/query/pattern-statement.md | 2 +- data-explorer/kusto/query/project-operator.md | 6 ++---- .../kusto/query/project-rename-operator.md | 6 +++--- .../kusto/query/project-reorder-operator.md | 14 ++++++++------ data-explorer/kusto/query/restrict-statement.md | 2 ++ data-explorer/kusto/query/set-statement.md | 1 + .../kusto/query/tabular-expression-statements.md | 4 +++- 8 files changed, 22 insertions(+), 17 deletions(-) diff --git a/data-explorer/kusto/query/let-statement.md b/data-explorer/kusto/query/let-statement.md index df7d3b2e60..db039372e0 100644 --- a/data-explorer/kusto/query/let-statement.md +++ b/data-explorer/kusto/query/let-statement.md @@ -3,7 +3,7 @@ title: Let statement description: Learn how to use the Let statement to set a variable name to define an expression or a function. ms.reviewer: alexans ms.topic: reference -ms.date: 08/11/2024 +ms.date: 01/28/2025 ms.localizationpriority: high --- # Let statement @@ -65,7 +65,7 @@ To optimize multiple uses of the `let` statement within a single query, see [Opt ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Define scalar values diff --git a/data-explorer/kusto/query/pattern-statement.md b/data-explorer/kusto/query/pattern-statement.md index d21b6850f6..c93378ff9e 100644 --- a/data-explorer/kusto/query/pattern-statement.md +++ b/data-explorer/kusto/query/pattern-statement.md @@ -60,7 +60,7 @@ For more information, see [Working with middle-tier applications](#work-with-mid ## Examples -[!INCLUDE [help-cluster](../includes/help-cluster-samples-stormevents.md)] +[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] In these examples, a pattern is defined. diff --git a/data-explorer/kusto/query/project-operator.md b/data-explorer/kusto/query/project-operator.md index 187d5caca1..ef31b5bbd3 100644 --- a/data-explorer/kusto/query/project-operator.md +++ b/data-explorer/kusto/query/project-operator.md @@ -65,7 +65,7 @@ StormEvents **Output** -The table shows only the top 10 results. +The table shows the first 10 results. |EventId|State|EventType| |--|--|--| @@ -98,7 +98,7 @@ StormEvents **Output** -The table shows only the first 10 results. +The table shows the first 10 results. |StartLocation| TotalInjuries| |--|--| @@ -114,9 +114,7 @@ The table shows only the first 10 results. |COLLIERVILLE| 6| |...|...| -::: moniker range="microsoft-fabric || azure-data-explorer || azure-monitor || microsoft-sentinel" ## Related content * [`extend`](extend-operator.md) * [series_stats](series-stats-function.md) -::: moniker-end \ No newline at end of file diff --git a/data-explorer/kusto/query/project-rename-operator.md b/data-explorer/kusto/query/project-rename-operator.md index 5baf3ec62b..e8b5bb118c 100644 --- a/data-explorer/kusto/query/project-rename-operator.md +++ b/data-explorer/kusto/query/project-rename-operator.md @@ -45,6 +45,6 @@ print a='a', b='b', c='c' **Output** -|new_a|new_b|c| -|---|---|---| -|a|b|c| +| new_a | new_b | c | +|--|--|--| +| a | b | c | diff --git a/data-explorer/kusto/query/project-reorder-operator.md b/data-explorer/kusto/query/project-reorder-operator.md index 863c2efc93..3bce4b2567 100644 --- a/data-explorer/kusto/query/project-reorder-operator.md +++ b/data-explorer/kusto/query/project-reorder-operator.md @@ -42,6 +42,8 @@ A table that contains columns in the order specified by the operator arguments. [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] +Review the examples and run them in your Data Explorer query page. + ### Reorder with b first Reorder a table with three columns (a, b, c) so the second column (b) will appear first. @@ -58,9 +60,9 @@ print a='a', b='b', c='c' **Output** -|b|a|c| -|---|---|---| -|b|a|c| +| b | a | c | +|--|--|--| +| b | a | c | ### Reorder with a first @@ -78,6 +80,6 @@ print b = 'b', a2='a2', a3='a3', a1='a1' **Output** -|a1|a2|a3|b| -|---|---|---|---| -|a1|a2|a3|b| +| a1 | a2 | a3 | b | +|--|--|--|--| +| a1 | a2 | a3 | b | diff --git a/data-explorer/kusto/query/restrict-statement.md b/data-explorer/kusto/query/restrict-statement.md index d0ac82e878..666cf0b0bc 100644 --- a/data-explorer/kusto/query/restrict-statement.md +++ b/data-explorer/kusto/query/restrict-statement.md @@ -40,6 +40,8 @@ The middle-tier application can prefix the user's query with a **logical model** [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] +Review the examples and run them in your Data Explorer query page. + ### Let statement The example uses a [let statement](let-statement.md) appearing before `restrict` statement. diff --git a/data-explorer/kusto/query/set-statement.md b/data-explorer/kusto/query/set-statement.md index 49d2095c28..94dd70c218 100644 --- a/data-explorer/kusto/query/set-statement.md +++ b/data-explorer/kusto/query/set-statement.md @@ -17,6 +17,7 @@ Request properties control how a query executes and returns results. They can be Request properties aren't formally a part of the Kusto Query Language and may be modified without being considered as a breaking language change. > [!NOTE] + > * To set request properties using [T-SQL](t-sql.md), see [Set request properties](t-sql.md#set-request-properties). > * To set request properties using the [Kusto client libraries](../api/client-libraries.md), see [Kusto Data ClientRequestProperties class](../api/netfx/about-kusto-data.md). diff --git a/data-explorer/kusto/query/tabular-expression-statements.md b/data-explorer/kusto/query/tabular-expression-statements.md index 4afdee8191..983fa60677 100644 --- a/data-explorer/kusto/query/tabular-expression-statements.md +++ b/data-explorer/kusto/query/tabular-expression-statements.md @@ -43,6 +43,8 @@ A tabular data source produces sets of records, to be further processed by tabul [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] +Review the examples and run them in your Data Explorer query page. + ### Filter rows by condition This query counts the number of records in the `StormEvents` table that have a value of "FLORIDA" in the `State` column. @@ -83,7 +85,7 @@ StormEvents **Output** | State | Population | TotalInjuries | -|---|---|---| +|--|--|--| | ALABAMA | 4918690 | 60 | | CALIFORNIA | 39562900 | 61 | | KANSAS | 2915270 | 63 | From 41bb62eae02b49137d20b2289b679bcffb38b81a Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Sun, 2 Feb 2025 13:57:02 +0200 Subject: [PATCH 07/27] Update data-explorer/kusto/management/data-export/export-data-to-storage.md Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- .../kusto/management/data-export/export-data-to-storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md index 83e7227a59..4d4968a8bb 100644 --- a/data-explorer/kusto/management/data-export/export-data-to-storage.md +++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md @@ -42,7 +42,7 @@ You must have at least [Database Viewer](../../access-control/role-based-access- | `fileExtension` | `string` | Indicates the "extension" part of the storage artifact (for example, `.csv` or `.tsv`). If compression is used, `.gz` is appended as well. | | `namePrefix` | `string` | Indicates a prefix to add to each generated storage artifact name. A random prefix is used if left unspecified. | | `encoding` | `string` | Indicates how to encode the text: `UTF8NoBOM` (default) or `UTF8BOM`. | -| `compressionType` | `string` | Indicates the type of compression to use. Possible values are `gzip`, and for `parquet` format `snappy`, `lz4_raw`, `brotli`, or `zstd` as well. Default is `gzip`. | +| `compressionType` | `string` | The type of compression to use. Possible values include `gzip`, and for Parquet format: `snappy`, `lz4_raw`, `brotli`, or `zstd`. Default is `gzip`. | | `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. | | `persistDetails` | `bool` | Indicates that the command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | | `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. | From 67fdf88ad6483503bfa6f8d69c78bb8ad4e28d29 Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Sun, 2 Feb 2025 13:57:31 +0200 Subject: [PATCH 08/27] Update data-explorer/kusto/management/external-tables-azure-storage.md Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/management/external-tables-azure-storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/external-tables-azure-storage.md b/data-explorer/kusto/management/external-tables-azure-storage.md index 659eb7e003..2d7e978f2e 100644 --- a/data-explorer/kusto/management/external-tables-azure-storage.md +++ b/data-explorer/kusto/management/external-tables-azure-storage.md @@ -122,7 +122,7 @@ external_table("ExternalTable") |------------------|----------|-------------------------------------------------------------------------------------| | `folder` | `string` | Table's folder | | `docString` | `string` | String documenting the table | -| `compressed` | `bool` | If set, indicates whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression will be auto detected based on the file) | +| `compressed` | `bool` | If set, indicates whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | | `compressionType` | `string` | If set, indicates the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression will be auto detected based on the file) | | `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), indicates whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). | | `namePrefix` | `string` | If set, indicates the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | From 7e4d2bc4ecdac18106cbb5b74f1e56d335ce7e4a Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Sun, 2 Feb 2025 14:17:15 +0200 Subject: [PATCH 09/27] Update data-explorer/kusto/management/external-tables-azure-storage.md Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/management/external-tables-azure-storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/external-tables-azure-storage.md b/data-explorer/kusto/management/external-tables-azure-storage.md index 2d7e978f2e..43d5f24543 100644 --- a/data-explorer/kusto/management/external-tables-azure-storage.md +++ b/data-explorer/kusto/management/external-tables-azure-storage.md @@ -123,7 +123,7 @@ external_table("ExternalTable") | `folder` | `string` | Table's folder | | `docString` | `string` | String documenting the table | | `compressed` | `bool` | If set, indicates whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | -| `compressionType` | `string` | If set, indicates the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression will be auto detected based on the file) | +| `compressionType` | `string` | If set, indicates the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | | `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), indicates whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). | | `namePrefix` | `string` | If set, indicates the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | | `fileExtension` | `string` | If set, indicates file extensions of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | From 5346e37b585846ff8a169b7f6be65e65996c35a9 Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Sun, 2 Feb 2025 14:18:00 +0200 Subject: [PATCH 10/27] Update data-explorer/kusto/management/show-external-tables.md Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/management/show-external-tables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/show-external-tables.md b/data-explorer/kusto/management/show-external-tables.md index 6149e0fa06..36f0b3edfb 100644 --- a/data-explorer/kusto/management/show-external-tables.md +++ b/data-explorer/kusto/management/show-external-tables.md @@ -39,7 +39,7 @@ You must have at least Database User, Database Viewer, Database Monitor to run t | TableType | `string` | Type of external table | | Folder | `string` | Table's folder | | DocString | `string` | String documenting the table | -| Properties | `string` | Table's JSON serialized properties (specific to the type of table; visit "Create or alter external table" for more information) | +| Properties | `string` | Table's JSON serialized properties (specific to the type of table; For more information, see [Create and alter Azure Storage external tables](external-tables-azure-storage.md) or [Create and alter delta external tables on Azure Storage](external-tables-delta-lake). | ## Example From f0d7d73c029f51975031756481dc19f204efdb92 Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Mon, 3 Feb 2025 11:51:58 +0200 Subject: [PATCH 11/27] Fix broken link in show-external-tables.md --- data-explorer/kusto/management/show-external-tables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/show-external-tables.md b/data-explorer/kusto/management/show-external-tables.md index 36f0b3edfb..60e8287519 100644 --- a/data-explorer/kusto/management/show-external-tables.md +++ b/data-explorer/kusto/management/show-external-tables.md @@ -39,7 +39,7 @@ You must have at least Database User, Database Viewer, Database Monitor to run t | TableType | `string` | Type of external table | | Folder | `string` | Table's folder | | DocString | `string` | String documenting the table | -| Properties | `string` | Table's JSON serialized properties (specific to the type of table; For more information, see [Create and alter Azure Storage external tables](external-tables-azure-storage.md) or [Create and alter delta external tables on Azure Storage](external-tables-delta-lake). | +| Properties | `string` | Table's JSON serialized properties (specific to the type of table; For more information, see [Create and alter Azure Storage external tables](external-tables-azure-storage.md) or [Create and alter delta external tables on Azure Storage](external-tables-delta-lake.md). | ## Example From ab04489148e2769e33cf6694e40be914f32e37fe Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Mon, 3 Feb 2025 10:05:41 +0000 Subject: [PATCH 12/27] add compression properties to external delta tables doc --- data-explorer/kusto/management/external-tables-delta-lake.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/data-explorer/kusto/management/external-tables-delta-lake.md b/data-explorer/kusto/management/external-tables-delta-lake.md index 85f2090ef6..84f742e8f9 100644 --- a/data-explorer/kusto/management/external-tables-delta-lake.md +++ b/data-explorer/kusto/management/external-tables-delta-lake.md @@ -58,6 +58,8 @@ The supported authentication methods are the same as those supported by [Azure S |------------------|----------|------------------------------------------------------------------------------------| | `folder` | `string` | Table's folder | | `docString` | `string` | String documenting the table | +| `compressed` | `bool` | If set, indicates whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | +| `compressionType` | `string` | If set, indicates the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | | `namePrefix` | `string` | If set, indicates the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | | `fileExtension` | `string` | If set, indicates file extensions of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | | `encoding` | `string` | Indicates how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | From 62bddf33db5d65e88c442933e0f3f51f052f9697 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Mon, 3 Feb 2025 17:31:42 +0200 Subject: [PATCH 13/27] Add introductory example sentences --- data-explorer/kusto/query/project-operator.md | 2 ++ .../kusto/query/project-rename-operator.md | 4 +++- .../kusto/query/project-reorder-operator.md | 4 ++-- .../kusto/query/restrict-statement.md | 2 ++ data-explorer/kusto/query/set-statement.md | 20 ++++++++++++++++++- .../query/tabular-expression-statements.md | 2 ++ 6 files changed, 30 insertions(+), 4 deletions(-) diff --git a/data-explorer/kusto/query/project-operator.md b/data-explorer/kusto/query/project-operator.md index ef31b5bbd3..cd0312c546 100644 --- a/data-explorer/kusto/query/project-operator.md +++ b/data-explorer/kusto/query/project-operator.md @@ -47,6 +47,8 @@ A table with columns that were named as arguments. Contains same number of rows ## Examples +The examples in this section show how to use the syntax to help you get started. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Only show specific columns diff --git a/data-explorer/kusto/query/project-rename-operator.md b/data-explorer/kusto/query/project-rename-operator.md index e8b5bb118c..4ab7a2ed8a 100644 --- a/data-explorer/kusto/query/project-rename-operator.md +++ b/data-explorer/kusto/query/project-rename-operator.md @@ -29,8 +29,10 @@ Renames columns in the output table. A table that has the columns in the same order as in an existing table, with columns renamed. -## Examples +## Example +The examples in this section show how to use the syntax to help you get started. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] :::moniker range="azure-data-explorer" diff --git a/data-explorer/kusto/query/project-reorder-operator.md b/data-explorer/kusto/query/project-reorder-operator.md index 3bce4b2567..8f36b3edd0 100644 --- a/data-explorer/kusto/query/project-reorder-operator.md +++ b/data-explorer/kusto/query/project-reorder-operator.md @@ -40,10 +40,10 @@ A table that contains columns in the order specified by the operator arguments. ## Examples +The examples in this section show how to use the syntax to help you get started. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] -Review the examples and run them in your Data Explorer query page. - ### Reorder with b first Reorder a table with three columns (a, b, c) so the second column (b) will appear first. diff --git a/data-explorer/kusto/query/restrict-statement.md b/data-explorer/kusto/query/restrict-statement.md index d0ac82e878..a4b424d07d 100644 --- a/data-explorer/kusto/query/restrict-statement.md +++ b/data-explorer/kusto/query/restrict-statement.md @@ -38,6 +38,8 @@ The middle-tier application can prefix the user's query with a **logical model** ## Examples +The examples in this section show how to use the syntax to help you get started. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Let statement diff --git a/data-explorer/kusto/query/set-statement.md b/data-explorer/kusto/query/set-statement.md index 94dd70c218..fe9576222d 100644 --- a/data-explorer/kusto/query/set-statement.md +++ b/data-explorer/kusto/query/set-statement.md @@ -36,9 +36,27 @@ Request properties aren't formally a part of the Kusto Query Language and may be ## Example +The example in this section shows how to use the syntax to help you get started. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] +:::moniker range="azure-data-explorer" +> [!div class="nextstepaction"] +> Run the query +::: moniker-end + ```kusto set querytrace; -Events | take 100 +StormEvents | take 100 ``` + +**Output** + +StartTime | EndTime |EpisodeId |EventId | State| EventType +|--|--|--|--|--|--| +2007-01-15T12:30:00Z | 2007-01-15T16:00:00Z | 1636 | 7821 | OHIO | Flood | +2007-08-03T01:50:00Z | 2007-08-03T01:50:00Z | 10085 | 56083 | NEW YORK | Thunderstorm Wind | +2007-08-03T15:33:00Z | 2007-08-03T15:33:00Z | 10086 | 56084 | NEW YORK | Hail | +2007-08-03T15:40:00Z | 2007-08-03T15:40:00Z | 10086 | 56085 | NEW YORK | Hail | +2007-08-03T23:15:00Z | 2007-08-05T04:30:00Z | 6569 | 38232 | NEBRASKA | Flood | +2007-08-06T18:19:00Z | 2007-08-06T18:19:00Z | 6719 | 39781 | IOWA | Thunderstorm Wind | diff --git a/data-explorer/kusto/query/tabular-expression-statements.md b/data-explorer/kusto/query/tabular-expression-statements.md index bef6ea9127..1acbe27dd9 100644 --- a/data-explorer/kusto/query/tabular-expression-statements.md +++ b/data-explorer/kusto/query/tabular-expression-statements.md @@ -41,6 +41,8 @@ A tabular data source produces sets of records, to be further processed by tabul ## Examples +The examples in this section show how to use the syntax to help you get started. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Filter rows by condition From e8e36e2160f71007790bde81a9079801be8bbb24 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Mon, 3 Feb 2025 17:37:46 +0200 Subject: [PATCH 14/27] Added sentence --- data-explorer/kusto/query/let-statement.md | 2 ++ data-explorer/kusto/query/set-statement.md | 3 +++ 2 files changed, 5 insertions(+) diff --git a/data-explorer/kusto/query/let-statement.md b/data-explorer/kusto/query/let-statement.md index db039372e0..f3c4cc1fab 100644 --- a/data-explorer/kusto/query/let-statement.md +++ b/data-explorer/kusto/query/let-statement.md @@ -65,6 +65,8 @@ To optimize multiple uses of the `let` statement within a single query, see [Opt ## Examples +The examples in this section show how to use the syntax to help you get started. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Define scalar values diff --git a/data-explorer/kusto/query/set-statement.md b/data-explorer/kusto/query/set-statement.md index fe9576222d..bed5f0ca9c 100644 --- a/data-explorer/kusto/query/set-statement.md +++ b/data-explorer/kusto/query/set-statement.md @@ -52,6 +52,8 @@ StormEvents | take 100 **Output** +The table shows the first few results. + StartTime | EndTime |EpisodeId |EventId | State| EventType |--|--|--|--|--|--| 2007-01-15T12:30:00Z | 2007-01-15T16:00:00Z | 1636 | 7821 | OHIO | Flood | @@ -60,3 +62,4 @@ StartTime | EndTime |EpisodeId |EventId | State| EventType 2007-08-03T15:40:00Z | 2007-08-03T15:40:00Z | 10086 | 56085 | NEW YORK | Hail | 2007-08-03T23:15:00Z | 2007-08-05T04:30:00Z | 6569 | 38232 | NEBRASKA | Flood | 2007-08-06T18:19:00Z | 2007-08-06T18:19:00Z | 6719 | 39781 | IOWA | Thunderstorm Wind | +|...|...|...|...|...|...| From 604280ec526b32e4027d396be3abc4e5d9b2bca5 Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Tue, 4 Feb 2025 10:10:39 +0200 Subject: [PATCH 15/27] Update data-explorer/kusto/management/data-export/export-data-to-storage.md --- .../kusto/management/data-export/export-data-to-storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md index 4d4968a8bb..92c924b07d 100644 --- a/data-explorer/kusto/management/data-export/export-data-to-storage.md +++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md @@ -42,7 +42,7 @@ You must have at least [Database Viewer](../../access-control/role-based-access- | `fileExtension` | `string` | Indicates the "extension" part of the storage artifact (for example, `.csv` or `.tsv`). If compression is used, `.gz` is appended as well. | | `namePrefix` | `string` | Indicates a prefix to add to each generated storage artifact name. A random prefix is used if left unspecified. | | `encoding` | `string` | Indicates how to encode the text: `UTF8NoBOM` (default) or `UTF8BOM`. | -| `compressionType` | `string` | The type of compression to use. Possible values include `gzip`, and for Parquet format: `snappy`, `lz4_raw`, `brotli`, or `zstd`. Default is `gzip`. | +| `compressionType` | `string` | The type of compression to use. Possible values include `gzip` and Parquet formats: `snappy`, `lz4_raw`, `brotli`, or `zstd`. Default is `gzip`. | | `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. | | `persistDetails` | `bool` | Indicates that the command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | | `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. | From b51cce43e4c7d93dd486cca94c90210afed0e405 Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Tue, 4 Feb 2025 10:12:01 +0200 Subject: [PATCH 16/27] Update data-explorer/kusto/management/external-tables-delta-lake.md --- data-explorer/kusto/management/external-tables-delta-lake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/external-tables-delta-lake.md b/data-explorer/kusto/management/external-tables-delta-lake.md index 84f742e8f9..6148837958 100644 --- a/data-explorer/kusto/management/external-tables-delta-lake.md +++ b/data-explorer/kusto/management/external-tables-delta-lake.md @@ -58,7 +58,7 @@ The supported authentication methods are the same as those supported by [Azure S |------------------|----------|------------------------------------------------------------------------------------| | `folder` | `string` | Table's folder | | `docString` | `string` | String documenting the table | -| `compressed` | `bool` | If set, indicates whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | +| `compressed` | `bool` | If set, files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | | `compressionType` | `string` | If set, indicates the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | | `namePrefix` | `string` | If set, indicates the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | | `fileExtension` | `string` | If set, indicates file extensions of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | From 4999ba678de288b5c85e0fad6c51095e78e69424 Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Tue, 4 Feb 2025 10:15:57 +0200 Subject: [PATCH 17/27] Apply suggestions from code review --- .../management/data-export/export-data-to-storage.md | 2 +- .../kusto/management/external-tables-azure-storage.md | 8 ++++---- .../kusto/management/external-tables-delta-lake.md | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md index 92c924b07d..5b383112b9 100644 --- a/data-explorer/kusto/management/data-export/export-data-to-storage.md +++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md @@ -44,7 +44,7 @@ You must have at least [Database Viewer](../../access-control/role-based-access- | `encoding` | `string` | Indicates how to encode the text: `UTF8NoBOM` (default) or `UTF8BOM`. | | `compressionType` | `string` | The type of compression to use. Possible values include `gzip` and Parquet formats: `snappy`, `lz4_raw`, `brotli`, or `zstd`. Default is `gzip`. | | `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. | -| `persistDetails` | `bool` | Indicates that the command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | +| `persistDetails` | `bool` | Specifies that the command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | | `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. | | `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. Default row group size is 100,000 records. | | `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. | diff --git a/data-explorer/kusto/management/external-tables-azure-storage.md b/data-explorer/kusto/management/external-tables-azure-storage.md index 43d5f24543..4131ce411d 100644 --- a/data-explorer/kusto/management/external-tables-azure-storage.md +++ b/data-explorer/kusto/management/external-tables-azure-storage.md @@ -123,10 +123,10 @@ external_table("ExternalTable") | `folder` | `string` | Table's folder | | `docString` | `string` | String documenting the table | | `compressed` | `bool` | If set, indicates whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | -| `compressionType` | `string` | If set, indicates the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | -| `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), indicates whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). | -| `namePrefix` | `string` | If set, indicates the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | -| `fileExtension` | `string` | If set, indicates file extensions of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | +| `compressionType` | `string` | If set, specifies the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | +| `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), specifies whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). | +| `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | +| `fileExtension` | `string` | If set, specifies the extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | | `encoding` | `string` | Indicates how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | | `sampleUris` | `bool` | If set, the command result provides several examples of simulated external data files URI as they're expected by the external table definition. This option helps validate whether the *Partitions* and *PathFormat* parameters are defined properly. | | `filesPreview` | `bool` | If set, one of the command result tables contains a preview of [.show external table artifacts](show-external-table-artifacts.md) command. Like `sampleUri`, the option helps validate the *Partitions* and *PathFormat* parameters of external table definition. | diff --git a/data-explorer/kusto/management/external-tables-delta-lake.md b/data-explorer/kusto/management/external-tables-delta-lake.md index 6148837958..3fb56b4db6 100644 --- a/data-explorer/kusto/management/external-tables-delta-lake.md +++ b/data-explorer/kusto/management/external-tables-delta-lake.md @@ -60,9 +60,9 @@ The supported authentication methods are the same as those supported by [Azure S | `docString` | `string` | String documenting the table | | `compressed` | `bool` | If set, files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | | `compressionType` | `string` | If set, indicates the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | -| `namePrefix` | `string` | If set, indicates the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | -| `fileExtension` | `string` | If set, indicates file extensions of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | -| `encoding` | `string` | Indicates how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | +| `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | +| `fileExtension` | `string` | If set, specifies extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | +| `encoding` | `string` | Specifies how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | | `dryRun` | `bool` | If set, the external table definition isn't persisted. This option is useful for validating the external table definition, especially in conjunction with the `filesPreview` or `sampleUris` parameter. | > [!NOTE] From 0428e157ba98fa9ec73800e3985e470079026bb7 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Tue, 4 Feb 2025 14:15:44 +0200 Subject: [PATCH 18/27] Fix note formatting in set-statement.md --- data-explorer/kusto/query/set-statement.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/set-statement.md b/data-explorer/kusto/query/set-statement.md index bed5f0ca9c..36b2598bcf 100644 --- a/data-explorer/kusto/query/set-statement.md +++ b/data-explorer/kusto/query/set-statement.md @@ -17,7 +17,7 @@ Request properties control how a query executes and returns results. They can be Request properties aren't formally a part of the Kusto Query Language and may be modified without being considered as a breaking language change. > [!NOTE] - +> > * To set request properties using [T-SQL](t-sql.md), see [Set request properties](t-sql.md#set-request-properties). > * To set request properties using the [Kusto client libraries](../api/client-libraries.md), see [Kusto Data ClientRequestProperties class](../api/netfx/about-kusto-data.md). From f8e651b4cafdae50dfbcfdcb8b856a89c10813c9 Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Tue, 4 Feb 2025 17:12:41 +0200 Subject: [PATCH 19/27] Clarify documentation for external table properties --- .../kusto/management/external-tables-azure-storage.md | 4 ++-- data-explorer/kusto/management/external-tables-delta-lake.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/data-explorer/kusto/management/external-tables-azure-storage.md b/data-explorer/kusto/management/external-tables-azure-storage.md index 4131ce411d..bffe1efdb9 100644 --- a/data-explorer/kusto/management/external-tables-azure-storage.md +++ b/data-explorer/kusto/management/external-tables-azure-storage.md @@ -122,12 +122,12 @@ external_table("ExternalTable") |------------------|----------|-------------------------------------------------------------------------------------| | `folder` | `string` | Table's folder | | `docString` | `string` | String documenting the table | -| `compressed` | `bool` | If set, indicates whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | +| `compressed` | `bool` | If set, specifies whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | | `compressionType` | `string` | If set, specifies the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | | `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), specifies whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). | | `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | | `fileExtension` | `string` | If set, specifies the extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | -| `encoding` | `string` | Indicates how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | +| `encoding` | `string` | Specifies how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | | `sampleUris` | `bool` | If set, the command result provides several examples of simulated external data files URI as they're expected by the external table definition. This option helps validate whether the *Partitions* and *PathFormat* parameters are defined properly. | | `filesPreview` | `bool` | If set, one of the command result tables contains a preview of [.show external table artifacts](show-external-table-artifacts.md) command. Like `sampleUri`, the option helps validate the *Partitions* and *PathFormat* parameters of external table definition. | | `validateNotEmpty` | `bool` | If set, the connection strings are validated for having content in them. The command will fail if the specified URI location doesn't exist, or if there are insufficient permissions to access it. | diff --git a/data-explorer/kusto/management/external-tables-delta-lake.md b/data-explorer/kusto/management/external-tables-delta-lake.md index 3fb56b4db6..bf73ca89fb 100644 --- a/data-explorer/kusto/management/external-tables-delta-lake.md +++ b/data-explorer/kusto/management/external-tables-delta-lake.md @@ -59,7 +59,7 @@ The supported authentication methods are the same as those supported by [Azure S | `folder` | `string` | Table's folder | | `docString` | `string` | String documenting the table | | `compressed` | `bool` | If set, files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | -| `compressionType` | `string` | If set, indicates the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | +| `compressionType` | `string` | If set, specifies the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | | `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | | `fileExtension` | `string` | If set, specifies extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | | `encoding` | `string` | Specifies how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | From bd5d56c7e63782bc22bbd1a189846259d8ac2323 Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Tue, 4 Feb 2025 17:33:57 +0200 Subject: [PATCH 20/27] Clarify descriptions in export-data-to-storage.md --- .../data-export/export-data-to-storage.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md index 5b383112b9..5e87717e4c 100644 --- a/data-explorer/kusto/management/data-export/export-data-to-storage.md +++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md @@ -27,8 +27,8 @@ You must have at least [Database Viewer](../../access-control/role-based-access- |--|--|--|--| | *async* | `string` | | If specified, the command runs in asynchronous mode. See [asynchronous mode](#asynchronous-mode). | | *compressed* | `bool` | | If specified, the output storage artifacts are compressed as `.gz` files. See the `compressionType` [supported property](#supported-properties) for compressing Parquet files as snappy. | -| *OutputDataFormat* | `string` | :heavy_check_mark: | Indicates the data format of the storage artifacts written by the command. Supported values are: `csv`, `tsv`, `json`, and `parquet`. | -| *StorageConnectionString* | `string` | | One or more [storage connection strings](../../api/connection-strings/storage-connection-strings.md) that indicate which storage to write the data to. More than one storage connection string might be specified for scalable writes. Each such connection string must indicate the credentials to use when writing to storage. For example, when writing to Azure Blob Storage, the credentials can be the storage account key, or a shared access key (SAS) with the permissions to read, write, and list blobs. | +| *OutputDataFormat* | `string` | :heavy_check_mark: | The data format of the storage artifacts written by the command. Supported values are: `csv`, `tsv`, `json`, and `parquet`. | +| *StorageConnectionString* | `string` | | One or more [storage connection strings](../../api/connection-strings/storage-connection-strings.md) that specify which storage to write the data to. More than one storage connection string might be specified for scalable writes. Each such connection string must specify the credentials to use when writing to storage. For example, when writing to Azure Blob Storage, the credentials can be the storage account key, or a shared access key (SAS) with the permissions to read, write, and list blobs. | | *PropertyName*, *PropertyValue* | `string` | | A comma-separated list of key-value property pairs. See [supported properties](#supported-properties).| > [!NOTE] @@ -39,16 +39,16 @@ You must have at least [Database Viewer](../../access-control/role-based-access- | Property | Type | Description | |--|--|--| | `includeHeaders` | `string` | For `csv`/`tsv` output, controls the generation of column headers. Can be one of `none` (default; no header lines emitted), `all` (emit a header line into every storage artifact), or `firstFile` (emit a header line into the first storage artifact only). | -| `fileExtension` | `string` | Indicates the "extension" part of the storage artifact (for example, `.csv` or `.tsv`). If compression is used, `.gz` is appended as well. | -| `namePrefix` | `string` | Indicates a prefix to add to each generated storage artifact name. A random prefix is used if left unspecified. | -| `encoding` | `string` | Indicates how to encode the text: `UTF8NoBOM` (default) or `UTF8BOM`. | +| `fileExtension` | `string` | The "extension" part of the storage artifact (for example, `.csv` or `.tsv`). If compression is used, `.gz` is appended as well. | +| `namePrefix` | `string` | The prefix to add to each generated storage artifact name. A random prefix is used if left unspecified. | +| `encoding` | `string` | The encoding for text. Possible values include: `UTF8NoBOM` (default) or `UTF8BOM`. | | `compressionType` | `string` | The type of compression to use. Possible values include `gzip` and Parquet formats: `snappy`, `lz4_raw`, `brotli`, or `zstd`. Default is `gzip`. | | `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. | -| `persistDetails` | `bool` | Specifies that the command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | +| `persistDetails` | `bool` | The command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | | `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. | | `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. Default row group size is 100,000 records. | | `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. | -| `parquetDatetimePrecision` | `string` | Specifies the precision to use when exporting `datetime` values to Parquet. Possible values are millisecond and microsecond. Default is millisecond. | +| `parquetDatetimePrecision` | `string` | The precision to use when exporting `datetime` values to Parquet. Possible values are millisecond and microsecond. Default is millisecond. | ## Authentication and authorization From 1dc96678fcaf0915051695db44c521d5122858f9 Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Tue, 4 Feb 2025 17:35:18 +0200 Subject: [PATCH 21/27] Clarify `persistDetails` description in documentation --- .../kusto/management/data-export/export-data-to-storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md index 5e87717e4c..c53e9a0578 100644 --- a/data-explorer/kusto/management/data-export/export-data-to-storage.md +++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md @@ -44,7 +44,7 @@ You must have at least [Database Viewer](../../access-control/role-based-access- | `encoding` | `string` | The encoding for text. Possible values include: `UTF8NoBOM` (default) or `UTF8BOM`. | | `compressionType` | `string` | The type of compression to use. Possible values include `gzip` and Parquet formats: `snappy`, `lz4_raw`, `brotli`, or `zstd`. Default is `gzip`. | | `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. | -| `persistDetails` | `bool` | The command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | +| `persistDetails` | `bool` | If `true`, the command persists its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | | `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. | | `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. Default row group size is 100,000 records. | | `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. | From 988144e3828f43f3959808446df6282b802a9e7c Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Wed, 5 Feb 2025 10:27:12 +0200 Subject: [PATCH 22/27] Update project-rename-operator.md example section --- data-explorer/kusto/query/project-rename-operator.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/data-explorer/kusto/query/project-rename-operator.md b/data-explorer/kusto/query/project-rename-operator.md index 4ab7a2ed8a..6fde356267 100644 --- a/data-explorer/kusto/query/project-rename-operator.md +++ b/data-explorer/kusto/query/project-rename-operator.md @@ -31,9 +31,7 @@ A table that has the columns in the same order as in an existing table, with col ## Example -The examples in this section show how to use the syntax to help you get started. - -[!INCLUDE [help-cluster](../includes/help-cluster-note.md)] +If you have a table with columns a, b, and c, and you want to rename a to new_a and b to new_b while keeping the same order, the query would look like this: :::moniker range="azure-data-explorer" > [!div class="nextstepaction"] From bd5563caf9c56f77a67084f99701be87c00849a3 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Thu, 6 Feb 2025 15:58:52 +0200 Subject: [PATCH 23/27] Update example description in set-statement.md --- data-explorer/kusto/query/set-statement.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/query/set-statement.md b/data-explorer/kusto/query/set-statement.md index 36b2598bcf..5334001f90 100644 --- a/data-explorer/kusto/query/set-statement.md +++ b/data-explorer/kusto/query/set-statement.md @@ -36,8 +36,8 @@ Request properties aren't formally a part of the Kusto Query Language and may be ## Example -The example in this section shows how to use the syntax to help you get started. - +This query enables query tracing and then fetches the first 100 records from the StormEvents table. + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] :::moniker range="azure-data-explorer" From 8a6ae1f86ea52fdabfc18547194912f4b3136711 Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Thu, 6 Feb 2025 16:01:50 +0200 Subject: [PATCH 24/27] conflict resolved --- data-explorer/kusto/query/set-statement.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/set-statement.md b/data-explorer/kusto/query/set-statement.md index 36b2598bcf..8aff876db1 100644 --- a/data-explorer/kusto/query/set-statement.md +++ b/data-explorer/kusto/query/set-statement.md @@ -36,7 +36,7 @@ Request properties aren't formally a part of the Kusto Query Language and may be ## Example -The example in this section shows how to use the syntax to help you get started. +This query enables query tracing and then fetches the first 100 records from the StormEvents table. [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] From a6a8a6c52dd3078a137f2e9d43dce4ded90e59d3 Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Sun, 9 Feb 2025 16:42:58 +0200 Subject: [PATCH 25/27] Apply suggestions from code review Co-authored-by: Slavik N <34075198+slavikn84@users.noreply.github.com> --- .../kusto/management/data-export/export-data-to-storage.md | 2 +- .../kusto/management/external-tables-azure-storage.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md index c53e9a0578..9b745d1eec 100644 --- a/data-explorer/kusto/management/data-export/export-data-to-storage.md +++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md @@ -42,7 +42,7 @@ You must have at least [Database Viewer](../../access-control/role-based-access- | `fileExtension` | `string` | The "extension" part of the storage artifact (for example, `.csv` or `.tsv`). If compression is used, `.gz` is appended as well. | | `namePrefix` | `string` | The prefix to add to each generated storage artifact name. A random prefix is used if left unspecified. | | `encoding` | `string` | The encoding for text. Possible values include: `UTF8NoBOM` (default) or `UTF8BOM`. | -| `compressionType` | `string` | The type of compression to use. Possible values include `gzip` and Parquet formats: `snappy`, `lz4_raw`, `brotli`, or `zstd`. Default is `gzip`. | +| `compressionType` | `string` | The type of compression to use. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. | | `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. | | `persistDetails` | `bool` | If `true`, the command persists its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | | `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. | diff --git a/data-explorer/kusto/management/external-tables-azure-storage.md b/data-explorer/kusto/management/external-tables-azure-storage.md index bffe1efdb9..1207cb7d9b 100644 --- a/data-explorer/kusto/management/external-tables-azure-storage.md +++ b/data-explorer/kusto/management/external-tables-azure-storage.md @@ -122,8 +122,8 @@ external_table("ExternalTable") |------------------|----------|-------------------------------------------------------------------------------------| | `folder` | `string` | Table's folder | | `docString` | `string` | String documenting the table | -| `compressed` | `bool` | If set, specifies whether the files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | -| `compressionType` | `string` | If set, specifies the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | +| `compressed` | `bool` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).
If set to true, the data is exported in the format specified by the `compressionType` property. For the read path, compression is automatically detected. | +| `compressionType` | `string` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).
The compression type of exported files. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. For the read path, compression type is automatically detected. | | `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), specifies whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). | | `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | | `fileExtension` | `string` | If set, specifies the extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | From 6c6021af8540af32faf7cbc27bc41bfea73ab649 Mon Sep 17 00:00:00 2001 From: Yael Epstein <108304247+yaela8@users.noreply.github.com> Date: Sun, 9 Feb 2025 17:03:09 +0000 Subject: [PATCH 26/27] align wording and table markdown --- .../data-export/export-data-to-storage.md | 38 +++++++++---------- .../external-tables-azure-storage.md | 26 ++++++------- .../management/external-tables-delta-lake.md | 20 +++++----- 3 files changed, 42 insertions(+), 42 deletions(-) diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md index 9b745d1eec..d5976ae2c5 100644 --- a/data-explorer/kusto/management/data-export/export-data-to-storage.md +++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md @@ -23,31 +23,31 @@ You must have at least [Database Viewer](../../access-control/role-based-access- ## Parameters -| Name | Type | Required | Description | -|--|--|--|--| -| *async* | `string` | | If specified, the command runs in asynchronous mode. See [asynchronous mode](#asynchronous-mode). | -| *compressed* | `bool` | | If specified, the output storage artifacts are compressed as `.gz` files. See the `compressionType` [supported property](#supported-properties) for compressing Parquet files as snappy. | -| *OutputDataFormat* | `string` | :heavy_check_mark: | The data format of the storage artifacts written by the command. Supported values are: `csv`, `tsv`, `json`, and `parquet`. | -| *StorageConnectionString* | `string` | | One or more [storage connection strings](../../api/connection-strings/storage-connection-strings.md) that specify which storage to write the data to. More than one storage connection string might be specified for scalable writes. Each such connection string must specify the credentials to use when writing to storage. For example, when writing to Azure Blob Storage, the credentials can be the storage account key, or a shared access key (SAS) with the permissions to read, write, and list blobs. | -| *PropertyName*, *PropertyValue* | `string` | | A comma-separated list of key-value property pairs. See [supported properties](#supported-properties).| +| Name | Type | Required | Description | +|-- |-- |-- |-- | +| *async* | `string` | | If specified, the command runs in asynchronous mode. See [asynchronous mode](#asynchronous-mode). | +| *compressed* | `bool` | | If specified, the output storage artifacts are compressed in the format specified by the `compressionType` [supported property](#supported-properties). | +| *OutputDataFormat* | `string` | :heavy_check_mark: | The data format of the storage artifacts written by the command. Supported values are: `csv`, `tsv`, `json`, and `parquet`. | +| *StorageConnectionString* | `string` | | One or more [storage connection strings](../../api/connection-strings/storage-connection-strings.md) that specify which storage to write the data to. More than one storage connection string might be specified for scalable writes. Each such connection string must specify the credentials to use when writing to storage. For example, when writing to Azure Blob Storage, the credentials can be the storage account key, or a shared access key (SAS) with the permissions to read, write, and list blobs. | +| *PropertyName*, *PropertyValue* | `string` | | A comma-separated list of key-value property pairs. See [supported properties](#supported-properties).| > [!NOTE] > We highly recommended exporting data to storage that is colocated in the same region as the database itself. This includes data that is exported so it can be transferred to another cloud service in other regions. Writes should be done locally, while reads can happen remotely. ## Supported properties -| Property | Type | Description | -|--|--|--| -| `includeHeaders` | `string` | For `csv`/`tsv` output, controls the generation of column headers. Can be one of `none` (default; no header lines emitted), `all` (emit a header line into every storage artifact), or `firstFile` (emit a header line into the first storage artifact only). | -| `fileExtension` | `string` | The "extension" part of the storage artifact (for example, `.csv` or `.tsv`). If compression is used, `.gz` is appended as well. | -| `namePrefix` | `string` | The prefix to add to each generated storage artifact name. A random prefix is used if left unspecified. | -| `encoding` | `string` | The encoding for text. Possible values include: `UTF8NoBOM` (default) or `UTF8BOM`. | -| `compressionType` | `string` | The type of compression to use. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. | -| `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. | -| `persistDetails` | `bool` | If `true`, the command persists its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. | -| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. | -| `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. Default row group size is 100,000 records. | -| `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. | +| Property | Type | Description | +|-- |-- |-- | +| `includeHeaders` | `string` | For `csv`/`tsv` output, controls the generation of column headers. Can be one of `none` (default; no header lines emitted), `all` (emit a header line into every storage artifact), or `firstFile` (emit a header line into the first storage artifact only). | +| `fileExtension` | `string` | The "extension" part of the storage artifact (for example, `.csv` or `.tsv`). If compression is used, `.gz` is appended as well. | +| `namePrefix` | `string` | The prefix to add to each generated storage artifact name. A random prefix is used if left unspecified. | +| `encoding` | `string` | The encoding for text. Possible values include: `UTF8NoBOM` (default) or `UTF8BOM`. | +| `compressionType` | `string` | The type of compression to use. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. | +| `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. | +| `persistDetails` | `bool` | If `true`, the command persists its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results. Defaults to `false` in synchronous executions, but can be turned on in those as well. | +| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. | +| `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. Default row group size is 100,000 records. | +| `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. | | `parquetDatetimePrecision` | `string` | The precision to use when exporting `datetime` values to Parquet. Possible values are millisecond and microsecond. Default is millisecond. | ## Authentication and authorization diff --git a/data-explorer/kusto/management/external-tables-azure-storage.md b/data-explorer/kusto/management/external-tables-azure-storage.md index 1207cb7d9b..0662196fa1 100644 --- a/data-explorer/kusto/management/external-tables-azure-storage.md +++ b/data-explorer/kusto/management/external-tables-azure-storage.md @@ -118,20 +118,20 @@ external_table("ExternalTable") ## Optional properties -| Property | Type | Description | -|------------------|----------|-------------------------------------------------------------------------------------| -| `folder` | `string` | Table's folder | -| `docString` | `string` | String documenting the table | -| `compressed` | `bool` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).
If set to true, the data is exported in the format specified by the `compressionType` property. For the read path, compression is automatically detected. | -| `compressionType` | `string` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).
The compression type of exported files. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. For the read path, compression type is automatically detected. | -| `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), specifies whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). | -| `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | -| `fileExtension` | `string` | If set, specifies the extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | -| `encoding` | `string` | Specifies how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | -| `sampleUris` | `bool` | If set, the command result provides several examples of simulated external data files URI as they're expected by the external table definition. This option helps validate whether the *Partitions* and *PathFormat* parameters are defined properly. | -| `filesPreview` | `bool` | If set, one of the command result tables contains a preview of [.show external table artifacts](show-external-table-artifacts.md) command. Like `sampleUri`, the option helps validate the *Partitions* and *PathFormat* parameters of external table definition. | +| Property | Type | Description | +|------------------ |----------|-------------------------------------------------------------------------------------| +| `folder` | `string` | Table's folder | +| `docString` | `string` | String documenting the table | +| `compressed` | `bool` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).
If set to true, the data is exported in the format specified by the `compressionType` property. For the read path, compression is automatically detected. | +| `compressionType` | `string` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).
The compression type of exported files. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. For the read path, compression type is automatically detected. | +| `includeHeaders` | `string` | For delimited text formats (CSV, TSV, ...), specifies whether files contain a header. Possible values are: `All` (all files contain a header), `FirstFile` (first file in a folder contains a header), `None` (no files contain a header). | +| `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | +| `fileExtension` | `string` | If set, specifies the extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | +| `encoding` | `string` | Specifies how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | +| `sampleUris` | `bool` | If set, the command result provides several examples of simulated external data files URI as they're expected by the external table definition. This option helps validate whether the *Partitions* and *PathFormat* parameters are defined properly. | +| `filesPreview` | `bool` | If set, one of the command result tables contains a preview of [.show external table artifacts](show-external-table-artifacts.md) command. Like `sampleUri`, the option helps validate the *Partitions* and *PathFormat* parameters of external table definition. | | `validateNotEmpty` | `bool` | If set, the connection strings are validated for having content in them. The command will fail if the specified URI location doesn't exist, or if there are insufficient permissions to access it. | -| `dryRun` | `bool` | If set, the external table definition isn't persisted. This option is useful for validating the external table definition, especially in conjunction with the `filesPreview` or `sampleUris` parameter. | +| `dryRun` | `bool` | If set, the external table definition isn't persisted. This option is useful for validating the external table definition, especially in conjunction with the `filesPreview` or `sampleUris` parameter. | > [!NOTE] > The external table isn't accessed during creation, only during query and export. Use the `validateNotEmpty` optional property during creation to make sure the table definition is valid and the storage is accessible. diff --git a/data-explorer/kusto/management/external-tables-delta-lake.md b/data-explorer/kusto/management/external-tables-delta-lake.md index bf73ca89fb..7946e08e5f 100644 --- a/data-explorer/kusto/management/external-tables-delta-lake.md +++ b/data-explorer/kusto/management/external-tables-delta-lake.md @@ -54,16 +54,16 @@ The supported authentication methods are the same as those supported by [Azure S ## Optional properties -| Property | Type | Description | -|------------------|----------|------------------------------------------------------------------------------------| -| `folder` | `string` | Table's folder | -| `docString` | `string` | String documenting the table | -| `compressed` | `bool` | If set, files are compressed as `.gz` files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | -| `compressionType` | `string` | If set, specifies the compression type of the files (used in [export scenario](data-export/export-data-to-an-external-table.md) only; read path compression is automatically detected based on the file format) | -| `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | -| `fileExtension` | `string` | If set, specifies extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | -| `encoding` | `string` | Specifies how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | -| `dryRun` | `bool` | If set, the external table definition isn't persisted. This option is useful for validating the external table definition, especially in conjunction with the `filesPreview` or `sampleUris` parameter. | +| Property | Type | Description | +|------------------ |---------- |------------------------------------------------------------------------------------| +| `folder` | `string` | Table's folder | +| `docString` | `string` | String documenting the table | +| `compressed` | `bool` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).
If set to true, the data is exported in the format specified by the `compressionType` property. For the read path, compression is automatically detected. | +| `compressionType` | `string` | Only relevant for the [export scenario](data-export/export-data-to-an-external-table.md).
The compression type of exported files. For non-Parquet files, only `gzip` is allowed. For Parquet files, possible values include `gzip`, `snappy`, `lz4_raw`, `brotli`, and `zstd`. Default is `gzip`. For the read path, compression type is automatically detected. | +| `namePrefix` | `string` | If set, specifies the prefix of the files. On write operations, all files will be written with this prefix. On read operations, only files with this prefix are read. | +| `fileExtension` | `string` | If set, specifies extension of the files. On write, files names will end with this suffix. On read, only files with this file extension will be read. | +| `encoding` | `string` | Specifies how the text is encoded: `UTF8NoBOM` (default) or `UTF8BOM`. | +| `dryRun` | `bool` | If set, the external table definition isn't persisted. This option is useful for validating the external table definition, especially in conjunction with the `filesPreview` or `sampleUris` parameter. | > [!NOTE] > The external delta table is accessed during creation, to infer the partitioning information and, optionally, the schema. Make sure that the table definition is valid and that the storage is accessible. From dc8a773cf84135ea1b314a5cd22fc7d5cb33631d Mon Sep 17 00:00:00 2001 From: ktalmor <193799742+ktalmor@users.noreply.github.com> Date: Mon, 10 Feb 2025 10:30:18 +0200 Subject: [PATCH 27/27] resolved conflict --- data-explorer/kusto/query/shuffle-query.md | 2 +- data-explorer/kusto/query/sort-operator.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/query/shuffle-query.md b/data-explorer/kusto/query/shuffle-query.md index d11cd7dfcf..cf41ce0a35 100644 --- a/data-explorer/kusto/query/shuffle-query.md +++ b/data-explorer/kusto/query/shuffle-query.md @@ -70,7 +70,7 @@ In some cases, the `hint.strategy = shuffle` is ignored, and the query won't run ## Examples The example in this section shows how to use the syntax to help you get started. - + [!INCLUDE [help-cluster](../includes/help-cluster-note.md)] ### Use summarize with shuffle diff --git a/data-explorer/kusto/query/sort-operator.md b/data-explorer/kusto/query/sort-operator.md index b7b746f236..4c67df5197 100644 --- a/data-explorer/kusto/query/sort-operator.md +++ b/data-explorer/kusto/query/sort-operator.md @@ -40,7 +40,7 @@ When the input table contains the special values `null`, `NaN`, `-inf` and `+inf |--|--|--| |**Nulls first**|`null`,`NaN`,`-inf`,`-5`,`0`,`5`,`+inf`|`null`,`NaN`,`+inf`,`5`,`0`,`-5`| |**Nulls last**|`-inf`,`-5`,`0`,`+inf`,`NaN`,`null`|`+inf`,`5`,`0`,`-5`,`NaN`,`null`| - + > [!NOTE] > > * Null and NaN values are always grouped together.