From a0ad08ce467fac1b7462cbb0269f87cb3509dc29 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 12:59:38 +0200 Subject: [PATCH 01/37] Create entity-group-alter.md --- .../kusto/management/entity-group-alter.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 data-explorer/kusto/management/entity-group-alter.md diff --git a/data-explorer/kusto/management/entity-group-alter.md b/data-explorer/kusto/management/entity-group-alter.md new file mode 100644 index 0000000000..fdc0e8c7f6 --- /dev/null +++ b/data-explorer/kusto/management/entity-group-alter.md @@ -0,0 +1,29 @@ +# .alter entity_group + +Alters an existing entity group and stores it inside the database metadata. + + +**Syntax** + +`.alter` `entity_group` [`ifnotexists`] *EntityGroupName* `(` EntityReference1 , ... `)` + +**Output** + +|Output parameter |Type |Description +|---|---|--- +|Name |String |The name of the entity group. +|Entities |String | An array of the entities defined into the entity group. + +> [!NOTE] +> * If the entity group doesn't exist, an error is returned. For creating a new entity group, see [.create entity_group](entity-group-create.md) +> * Requires [database admin permission](../management/access-control/role-based-authorization.md) + +**Examples** + +```kusto +.alter entity_group MyEntityGroup (cluster('c1').database('d1')) +``` + +|Name|Entities| +|---|---| +|MyEntityGroup|["cluster('c1').database('d1')"]| From ad34ef21b7d2286db53acb1f43c8aa07f2c94641 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:02:53 +0200 Subject: [PATCH 02/37] Create entity-group-alter-merge.md --- .../management/entity-group-alter-merge.md | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 data-explorer/kusto/management/entity-group-alter-merge.md diff --git a/data-explorer/kusto/management/entity-group-alter-merge.md b/data-explorer/kusto/management/entity-group-alter-merge.md new file mode 100644 index 0000000000..35fe6409c8 --- /dev/null +++ b/data-explorer/kusto/management/entity-group-alter-merge.md @@ -0,0 +1,43 @@ +# .alter-merge entity_group + +Alters and merges an existing entity group with the provided list of entities and stores it inside the database metadata. + + +**Syntax** + +`.alter-merge` `entity_group` *EntityGroupName* `(` EntityReference1 , ... `)` + +**Output** + +|Output parameter |Type |Description +|---|---|--- +|Name |String |The name of the entity group. +|Entities |String | An array of the entities defined into the entity group. + +> [!NOTE] +> * If the entity group doesn't exist, an error is returned. For creating a new entity group, see [.create entity_group](entity-group-create.md) +> * Requires [database admin permission](../management/access-control/role-based-authorization.md) + +**Examples** + +Running the following command to create a new entity group: + +```kusto +.create entity_group MyEntityGroup (cluster('c1').database('d1')) +``` + +|Name|Entities| +|---|---| +|MyEntityGroup|["cluster('c1').database('d1')"]| + + +Then running the following command to edit the existing entity group MyEntityGroup and add the entity `cluster('c2').database('d2')` : + +```kusto +.alter-merge entity_group MyEntityGroup (cluster('c2').database('d2')) +``` + +|Name|Entities| +|---|---| +|MyEntityGroup|["cluster('c1').database('d1')","cluster('c2').database('d2')"]| + From a4cde2457083d6d45df333b5cb984b4f47f29783 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:04:38 +0200 Subject: [PATCH 03/37] Create entity-group-create.md --- .../kusto/management/entity-group-create.md | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 data-explorer/kusto/management/entity-group-create.md diff --git a/data-explorer/kusto/management/entity-group-create.md b/data-explorer/kusto/management/entity-group-create.md new file mode 100644 index 0000000000..bc32db32d3 --- /dev/null +++ b/data-explorer/kusto/management/entity-group-create.md @@ -0,0 +1,33 @@ +# .create entity_group + +Creates a stored entity group which is a reusable [`let` statement](../query/letstatement.md) +entity group with the given name. The entity group definition is persisted with the database metadata. + + +**Syntax** + +`.create` `entity_group` [`ifnotexists`] *EntityGroupName* `(` EntityReference1 , ... `)` + +**Output** + +|Output parameter |Type |Description +|---|---|--- +|Name |String |The name of the entity group. +|Entities |String | An array of the entities defined into the entity group. + +> [!NOTE] +> * If entity group already exists: +> * If `ifnotexists` flag is specified, the command is ignored (no change applied). +> * If `ifnotexists` flag is NOT specified, an error is returned. +> * For altering an existing entity group, see [.alter entity_group](entity-group-alter.md) +> * Requires [database admin permission](../management/access-control/role-based-authorization.md). + +**Examples** + +```kusto +.create entity_group MyEntityGroup (cluster('c1').database('d1'), cluster('c2').database('d2')) +``` + +|Name|Entities| +|---|---| +|MyEntityGroup|["cluster('c1').database('d1')","cluster('c2').database('d2')"]| From 06b25ad7345b6df22441e220c83d2a589dd99348 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:05:25 +0200 Subject: [PATCH 04/37] Create entity-group-drop.md --- .../kusto/management/entity-group-drop.md | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 data-explorer/kusto/management/entity-group-drop.md diff --git a/data-explorer/kusto/management/entity-group-drop.md b/data-explorer/kusto/management/entity-group-drop.md new file mode 100644 index 0000000000..30f539ba76 --- /dev/null +++ b/data-explorer/kusto/management/entity-group-drop.md @@ -0,0 +1,22 @@ +# .drop entity_group + +Drops an entity group from the database. + +**Syntax** + +`.drop` `entity_group` *EntityGroupName* + +**Output** + +This command returns a list of the remaining tables in the database. + +|Output parameter |Type |Description +|---|---|--- +|Name |String |The name of the entity group. +|Entities |String | An array of the entities defined into the entity group. + +**Examples** + +```kusto +.drop entity_group MyEntityGroup +``` From e92cb3423b8d92ef3ed4a93061a664f4374b0aca Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:06:32 +0200 Subject: [PATCH 05/37] Create show-entity-group.md --- .../kusto/management/show-entity-group.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 data-explorer/kusto/management/show-entity-group.md diff --git a/data-explorer/kusto/management/show-entity-group.md b/data-explorer/kusto/management/show-entity-group.md new file mode 100644 index 0000000000..42cc94d405 --- /dev/null +++ b/data-explorer/kusto/management/show-entity-group.md @@ -0,0 +1,59 @@ +# .show entity_group(s) + +Lists all the [stored entity groups](entity-groups.md) in the currently-selected database. +To return only one specific entity group, see [.show entity_group](#show-entity_group). + +## .show entity_groups + +```kusto +.show entity_groups +``` + +Requires [database user permission](../management/access-control/role-based-authorization.md). + +|Output parameter |Type |Description +|---|---|--- +|Name |String |The name of the entity group. +|Entities|String|The value of the entity group as an array that includes the entities. + +**Output example** + +|Name|Entities| +|---|---| +|eg1|["cluster('c1').database('d1')"]| +|eg2|["cluster('c2').database('d2')"]| + + +## .show entity_group + +```kusto +.show entity_group MyEntityGroup1 +``` + +Lists the details of one specific stored entity group. +For a list of **all** entity groups, see [.show entity_groups](#show-entity_groups). + +**Syntax** + +`.show` `entity_group` *EntityGroupName* + +**Output** + +|Output parameter |Type |Description +|---|---|--- +|Name |String |The name of the entity group. +|Entities|String|The value of the entity group as an array that includes the entities. + +> [!NOTE] +> * If the entity group does not exist, an error is returned. +> * Requires [database user permission](../management/access-control/role-based-authorization.md). + +**Example** + +```kusto +.show entity_group eg1 +``` + +|Name|Entities| +|---|---| +|eg1|["cluster('c1').database('d1')"]| From a65c900073c255b4a7de5f8f552e156ba7f4698d Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:07:32 +0200 Subject: [PATCH 06/37] Create entity-groups.md --- data-explorer/kusto/management/entity-groups.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 data-explorer/kusto/management/entity-groups.md diff --git a/data-explorer/kusto/management/entity-groups.md b/data-explorer/kusto/management/entity-groups.md new file mode 100644 index 0000000000..eb1149f41b --- /dev/null +++ b/data-explorer/kusto/management/entity-groups.md @@ -0,0 +1,17 @@ +# Stored entity group management overview + +Stored entity groups are named entities that are stored in a database and can be referenced by the [macro-expand query operator](../query/macro-expand-operator.md). +Storing an entity group in the database instead of providing its value in the query text itself makes it easier to manage these objects. + +> [!NOTE] +> A query can only reference entity groups defined in the query text or in the database-in-context. Entity groups in other databases or clusters (directly or indirectly) cannot be referenced. + +This section describes control commands used for creating, altering and dropping entity groups: + +|Function |Description| +|---------|-----------| +|[.alter entity_group](entity-group-alter.md) |Alters an existing entity group and stores it inside the database metadata | +|[.alter-merge entity_group](entity-group-alter-merge.md) |Alters and merge the value of an existing function | +|[.create entity_group](entity-group-create.md) |Creates a stored entity group| +|[.drop entity_grop](entity-group-drop.md) |Drops an entity group from the database | +|[.show entity_grop and .show entity_groups](entity-group-show.md) |Lists all the stored entity groups, or a specific entity group, in the currently-selected database| From d6bdcf1afe3ab8e62bd6d896202e9f0b80cf2191 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:08:27 +0200 Subject: [PATCH 07/37] Rename entity-group-alter-merge.md to alter-merge-entity-group.md --- .../{entity-group-alter-merge.md => alter-merge-entity-group.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename data-explorer/kusto/management/{entity-group-alter-merge.md => alter-merge-entity-group.md} (100%) diff --git a/data-explorer/kusto/management/entity-group-alter-merge.md b/data-explorer/kusto/management/alter-merge-entity-group.md similarity index 100% rename from data-explorer/kusto/management/entity-group-alter-merge.md rename to data-explorer/kusto/management/alter-merge-entity-group.md From 1a6e7dd4daa3588220b534225a3a56392fef6d1b Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:08:49 +0200 Subject: [PATCH 08/37] Rename entity-group-alter.md to alter-entity-group.md --- .../management/{entity-group-alter.md => alter-entity-group.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename data-explorer/kusto/management/{entity-group-alter.md => alter-entity-group.md} (100%) diff --git a/data-explorer/kusto/management/entity-group-alter.md b/data-explorer/kusto/management/alter-entity-group.md similarity index 100% rename from data-explorer/kusto/management/entity-group-alter.md rename to data-explorer/kusto/management/alter-entity-group.md From 932d500a6d01e03f38c8c5ef08e25a5a3c5d41eb Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:09:22 +0200 Subject: [PATCH 09/37] Rename entity-group-create.md to create-entity-group.md --- .../management/{entity-group-create.md => create-entity-group.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename data-explorer/kusto/management/{entity-group-create.md => create-entity-group.md} (100%) diff --git a/data-explorer/kusto/management/entity-group-create.md b/data-explorer/kusto/management/create-entity-group.md similarity index 100% rename from data-explorer/kusto/management/entity-group-create.md rename to data-explorer/kusto/management/create-entity-group.md From 4b70c77347e2d44563d67022dc0cb618c0ffd0c5 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:09:52 +0200 Subject: [PATCH 10/37] Rename entity-group-drop.md to drop-entity-group.md --- .../management/{entity-group-drop.md => drop-entity-group.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename data-explorer/kusto/management/{entity-group-drop.md => drop-entity-group.md} (100%) diff --git a/data-explorer/kusto/management/entity-group-drop.md b/data-explorer/kusto/management/drop-entity-group.md similarity index 100% rename from data-explorer/kusto/management/entity-group-drop.md rename to data-explorer/kusto/management/drop-entity-group.md From 145d60e7a39e68f5aa8ae8efe6499ca3b3495e22 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:10:57 +0200 Subject: [PATCH 11/37] Update entity-groups.md --- data-explorer/kusto/management/entity-groups.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/data-explorer/kusto/management/entity-groups.md b/data-explorer/kusto/management/entity-groups.md index eb1149f41b..f751dee5ef 100644 --- a/data-explorer/kusto/management/entity-groups.md +++ b/data-explorer/kusto/management/entity-groups.md @@ -10,8 +10,8 @@ This section describes control commands used for creating, altering and dropping |Function |Description| |---------|-----------| -|[.alter entity_group](entity-group-alter.md) |Alters an existing entity group and stores it inside the database metadata | -|[.alter-merge entity_group](entity-group-alter-merge.md) |Alters and merge the value of an existing function | -|[.create entity_group](entity-group-create.md) |Creates a stored entity group| -|[.drop entity_grop](entity-group-drop.md) |Drops an entity group from the database | -|[.show entity_grop and .show entity_groups](entity-group-show.md) |Lists all the stored entity groups, or a specific entity group, in the currently-selected database| +|[.alter entity_group](alter-entity-group.md) |Alters an existing entity group and stores it inside the database metadata | +|[.alter-merge entity_group](alter-merge-entity-group.md) |Alters and merge the value of an existing function | +|[.create entity_group](create-entity-group.md) |Creates a stored entity group| +|[.drop entity_grop](drop-entity-group.md) |Drops an entity group from the database | +|[.show entity_grop and .show entity_groups](show-entity-group.md) |Lists all the stored entity groups, or a specific entity group, in the currently-selected database| From 8e0e33a2e949b67cccb8b0b06b4b2ffcc9e70029 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:18:28 +0200 Subject: [PATCH 12/37] Update toc.yml --- data-explorer/kusto/management/toc.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/data-explorer/kusto/management/toc.yml b/data-explorer/kusto/management/toc.yml index 74767c809f..2fbf5691cb 100644 --- a/data-explorer/kusto/management/toc.yml +++ b/data-explorer/kusto/management/toc.yml @@ -253,6 +253,20 @@ items: - name: .alter-merge table column-docstrings command displayName: .alter-merge column-docstrings href: alter-merge-table-column.md + - name: Entity Group + items: + - name: Entity Group + href: entity-groups.md + - name: .create entity_group command + href: create-entity-group.md + - name: .alter entity_group command + href: alter-entity-group.md + - name: .alter-merge entity_group command + href: alter-merge-entity-group.md + - name: .drop entity_group command + href: drop-entity-group.md + - name: .show entity_group command + href: show-entity-group.md - name: Ingestion mappings items: - name: Ingestion mappings overview From 0d563dcf5dcc48592ce22841cb61978eb8bcd2cc Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:24:01 +0200 Subject: [PATCH 13/37] Create macro-expand-operator.md --- .../kusto/query/macro-expand-operator.md | 199 ++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 data-explorer/kusto/query/macro-expand-operator.md diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md new file mode 100644 index 0000000000..2189713600 --- /dev/null +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -0,0 +1,199 @@ +# The macro-expand operator + +The `macro-expand` operator provides concise syntax for applying a subquery +to a set of entities (clusters, databases, or tables) and combining their results. +It may come handy, for example, when one has data spread over multiple +clusters/databases/table (for example, when data is held in the same global region +as its source). The set of entities is called **entity group**; it can be stored for +reuse in a database, or defined ad-hoc in the query text itself. + +> [!NOTE] +> The operation of the `macro-expand` operator can be altered by setting the `best_effort` request property to `true`, using either a `set statement` or through [client request properties](../api/netfx/request-properties.md). When this property is set to `true`, the `macro-expand` operator will disregard fuzzy resolution and connectivity failures to execute any of the sub-expressions being “unioned” and yield a warning in the query status results. + +## Computation model + +The `macro-expand` operator takes three arguments: + +1. An identifier whose appearance in the subquery is expanded + into each of the entities in the entity group. + +1. An entity group, which is a collection of entity references; + these can be references to clusters, to databases, or to tables. + (The same entity group can only reference entities of the same kind, + however; one can't mix clusters and databases in the same entity group, + for example.) + +1. A subquery that takes no tabular input, produces a single tabular output, + and may reference entities in the entity group through the use of the + aforementioned identifier. The subquery must include exactly one tabular + expression, and may optionally include additional let-statements preceding + the tabular expression. + +The computation model of the `macro-expand` operator is as follows: +The operator takes no input, and is the `union` of expanding the subquery +as many times as there are individual entities in the entity group. + +## Syntax + +There are several syntactic variants one may use with the `macro-expand` operator: + +1. Explicit: In this syntax, the operator name (`macro-expand`), identifier, + entity group, and subquery are all explicit in the text of the operator + invocation itself. + +2. Explicit using let statement: This syntax is similar to the explicit syntax, + with the difference being that the entity group is specified in the query + using a let statement (that is, outside the `macro-expand` operator). + +3. Explicit using stored entity group: This syntax is similar to the syntax + that uses the let statement, only the query itself doesn't define the entity group. + Rather, the group is stored in the database-in-scope as a named entity. + +### Explicit syntax + +`macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] *EntityGroup* `as` *EntityIdentifier* `(` *Subquery* `)` + +Where: + +* *EntityGroup* has the syntax `entity_group` `[` *EntityReference1* [`,` ...] `]`, + and denotes one or more entities that *EntityIdentifier* expands to. + +> [!NOTE] +> A query can only reference entity groups defined in the query text or in the database-in-context. Entity groups in other databases or clusters (directly or indirectly) cannot be referenced. + +* *EntityIdentifier* is an identifier whose appearance in *Subquery* + gets expanded into each of the entities in the group. + +* *Subquery* is a tabular expression (optionally preceded by one or more let + statements) that takes no source and may reference *EntityIdentifier*. + All entity references other than *EntityIndentifier* behave "normally"; they + reference other query let statements, stored functions, etc. from the + database-in-scope. + +* *Kind* can take one of two values: + - `inner` - The result has the subset of columns that are common to all of the accessed scoped entities. + - `outer` - The result has all the columns that occur in any of the inputs. Cells that weren't defined by an input row are set to null. This is the default. + +* *IsFuzzy* (is set to `true`) enables fuzzy resolution of macro-expand scoped entities. + Fuzzy applies to the set of union sources. It means that while analyzing the query and preparing for execution, + the set of scope sources is reduced to the set of entities that exist and are accessible at the time. + If at least one such entity was found, any resolution failure will yield a warning in the query status results + (one for each missing reference), but will not prevent the query execution; + if no resolutions were successful - the query will return an error. The default is `false`. + +For example, the query below calculates the number of errors produced by each SKU. + +```kusto +macro-expand entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')] as X +( + X.KustoLogs + | where Level == 'Error' + | summarize Count=count() by Source + | join kind=inner (X.DimCluster | project SKU, Source) on Source +) +| summarize Sum=sum(Count) by SKU +``` + +Here's another way to write the same query without using `macro-expand`: + +```kusto +union + ( + cluster('C1').database('Kuskus').KustoLogs + | where Level == 'Error' + | summarize Count=count() by Source + | join kind=inner (cluster('C1').database('Kuskus').DimCluster | project SKU, Source) on Source + ), + ( + cluster('C2').database('Kuskus').KustoLogs + | where Level == 'Error' + | summarize Count=count() by Source + | join kind=inner (cluster('C2').database('Kuskus').DimCluster | project SKU, Source) on Source + ) +| summarize Sum=sum(Count) by SKU +``` + +In this example: + +1. The `entity_group` is a collection of databases from different clusters. + +1. The identifier `X` is used to scope entity references. Entities that are + are not scoped by this identifier are taken to be in the database-in-scope + scope (if they are not overridden as query-provided identifiers.) + +1. The `summarize` operator operates on the `union` of all expanded subqueries. + As the `expand` operator get applied only to the subquery between the `(` and `)` + characters, it's clear which scope is expanded and which is not. + +1. The `join` operator above is calculated two times, one for each entity in the + `entity_group`. Also note that `join` here is being done between two tables + of the same entity (denoted by `X`) &emdash; there's no cross-cluster `join` here. + +1. Not shown in this example, the subquery part can include additional uses of + the `macro-expand` operator (nested expansions). Note that the identifier of the inner + `macro-expand` cannot equal that of the outer, providing a clear way to distinguish the + scope. + +### Special Contextual functions + +The `macro-expand` subquery can reference two specialized scalar functions as-if they are part of the entity being referenced: + +1. `$current_database` - Returns the database name of the entity reference. +2. `$current_cluster_endpoint` - Returns the URL of the cluster of the entity reference. + +Examples: + +```kusto +macro-expand MyEntityGroup as X +( + X.Admins + | extend current_database = X.$current_database, current_cluster = X.$current_cluster_endpoint +) +| summarize count() by current_cluster, current_database +``` + +```kusto +macro-expand MyEntityGroup_Outer as X +( + macro-expand MyEntityGroup_Inner as Y + ( + X.Admins + | where Timestamp > ago(1h) + | extend lhs_database = X.$current_database, lhs_cluster = X.$current_cluster_endpoint + | join ( + Y.Admins + | where Timestamp > ago(1h) + | extend rhs_database = Y.$current_database, rhs_cluster = Y.$current_cluster_endpoint + ) on Source + ) +) +| summarize count() by lhs_cluster, lhs_database, rhs_cluster, rhs_database +``` + +### Explicit using let statement + +`macro-expand` *EntityIdentifier* `in` *EntityGroupIdentifier* `(` *Subquery* `)` + +Here, *EntityGroupIdentifier* is simply an identifier which has been defined by +the query using a let statement: + +`let` *EntityGroupIdentifier* `=` `entity_group` `[` *EntityReference1* [`,` ...] `]` + +For example, the previous example could be re-written as: + +```kusto +let UberKuskus = entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')]; +macro-expand UberKuskus as X +( + X.KustoLogs + | where Level == 'Error' + | summarize Count=count() by Source + | join kind=inner (X.DimCluster | project SKU, Source) on Source +) +| summarize Sum=sum(Count) by SKU +``` + +### Explicit using stored entity group + +The syntax is exactly the same as for using the let statement. +See here [Stored Entity Groups Management](../management/entity-groups.md) From 0aae5e8120e151413108220e86edc1eddbddaca7 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:25:06 +0200 Subject: [PATCH 14/37] Update toc.yml --- data-explorer/kusto/query/toc.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/toc.yml b/data-explorer/kusto/query/toc.yml index a5cff9d2a8..6092c4bb57 100644 --- a/data-explorer/kusto/query/toc.yml +++ b/data-explorer/kusto/query/toc.yml @@ -167,6 +167,8 @@ items: href: join-time-window.md - name: lookup operator href: lookup-operator.md + - name: macro-expand operator + href: macro-expand.md - name: mv-apply operator href: mv-apply-operator.md displayName: mv apply mv_apply @@ -1682,4 +1684,4 @@ items: - name: Timezones href: timezone.md - name: KQL docs navigation - href: ../docs-navigation.md \ No newline at end of file + href: ../docs-navigation.md From 2ecbc39671543ca69ecb97190c8caed15fa0ae81 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:27:51 +0200 Subject: [PATCH 15/37] Update macro-expand-operator.md --- data-explorer/kusto/query/macro-expand-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 2189713600..f664200a90 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -8,7 +8,7 @@ as its source). The set of entities is called **entity group**; it can be stored reuse in a database, or defined ad-hoc in the query text itself. > [!NOTE] -> The operation of the `macro-expand` operator can be altered by setting the `best_effort` request property to `true`, using either a `set statement` or through [client request properties](../api/netfx/request-properties.md). When this property is set to `true`, the `macro-expand` operator will disregard fuzzy resolution and connectivity failures to execute any of the sub-expressions being “unioned” and yield a warning in the query status results. +> The operation of the `macro-expand` operator can be altered by setting the `best_effort` request property to `true`, using either a `set statement` or through [client request properties](../api/rest/request-properties.md). When this property is set to `true`, the `macro-expand` operator will disregard fuzzy resolution and connectivity failures to execute any of the sub-expressions being “unioned” and yield a warning in the query status results. ## Computation model From 28a3b4faa2ad8a82248198a6c3109a68c31c0167 Mon Sep 17 00:00:00 2001 From: ziham1531991 <31045462+ziham1531991@users.noreply.github.com> Date: Mon, 13 Jan 2025 13:36:28 +0200 Subject: [PATCH 16/37] Update toc.yml --- data-explorer/kusto/query/toc.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/toc.yml b/data-explorer/kusto/query/toc.yml index 6092c4bb57..121ee7ffab 100644 --- a/data-explorer/kusto/query/toc.yml +++ b/data-explorer/kusto/query/toc.yml @@ -168,7 +168,7 @@ items: - name: lookup operator href: lookup-operator.md - name: macro-expand operator - href: macro-expand.md + href: macro-expand-operator.md - name: mv-apply operator href: mv-apply-operator.md displayName: mv apply mv_apply From ef68ae3a6fcd4a3772dc0f1717bce634d7ebc1d1 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Mon, 20 Jan 2025 17:19:39 +0200 Subject: [PATCH 17/37] entity groups and macro-expand --- .../kusto/management/alter-entity-group.md | 65 ++++-- .../management/alter-merge-entity-group.md | 67 ++++-- .../kusto/management/create-entity-group.md | 64 ++++-- .../kusto/management/drop-entity-group.md | 49 ++++- .../kusto/management/entity-groups.md | 36 +++- .../kusto/management/show-entity-group.md | 91 +++++---- .../kusto/query/macro-expand-operator.md | 192 ++++++++---------- 7 files changed, 358 insertions(+), 206 deletions(-) diff --git a/data-explorer/kusto/management/alter-entity-group.md b/data-explorer/kusto/management/alter-entity-group.md index fdc0e8c7f6..bb35b74206 100644 --- a/data-explorer/kusto/management/alter-entity-group.md +++ b/data-explorer/kusto/management/alter-entity-group.md @@ -1,29 +1,64 @@ -# .alter entity_group +--- +title: .alter entity_group +description: Learn how to use the `.alter entity_group` command to change an existing entity group. +ms.reviewer: ziham1531991 +ms.topic: reference +ms.date: 01/19/2025 +--- -Alters an existing entity group and stores it inside the database metadata. - -**Syntax** +# .alter entity_group -`.alter` `entity_group` [`ifnotexists`] *EntityGroupName* `(` EntityReference1 , ... `)` +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] -**Output** - -|Output parameter |Type |Description -|---|---|--- -|Name |String |The name of the entity group. -|Entities |String | An array of the entities defined into the entity group. +Alters an existing entity group and stores it inside the database metadata. For more information, see [Entity groups](entity-groups.md). -> [!NOTE] -> * If the entity group doesn't exist, an error is returned. For creating a new entity group, see [.create entity_group](entity-group-create.md) -> * Requires [database admin permission](../management/access-control/role-based-authorization.md) +## Permissions -**Examples** +You must have at least [Database Admin](../access-control/role-based-access-control.md) permissions to run this command. + +## Syntax + +`.alter` `entity_group` [`ifnotexists`] *EntityGroupName* `(`[*EntityReference*`,` ...]`)` + +[!INCLUDE [syntax-conventions-note](../includes/syntax-conventions-note.md)] + +## Parameters + +|Name|Type|Required|Description| +|--|--|--|--| +| `ifnotexists` | `string` | | If specified, the entity group only is created if the entity group doesn't exist yet.| +|*EntityGroupName*| `string` | :heavy_check_mark:|The name of the entity group. | +|*EntityReference*| `string` | :heavy_check_mark:|An entity included in the entity group. | + +## Returns + +This command returns a table with the following columns: + +|Output parameter |Type |Description| +|---|---|---| +|Name | `string` | The name of the entity group.| +|Entities | `array` | An array which includes one or more entities. If the entity group doesn't exist, and the `ifnotexists` flag isn't specified, an error is returned.| + +## Examples + +The following example alters `MyEntityGroup` to include the entity `cluster('c1').database('d1')`. ```kusto .alter entity_group MyEntityGroup (cluster('c1').database('d1')) ``` +**Output** + |Name|Entities| |---|---| |MyEntityGroup|["cluster('c1').database('d1')"]| + +## Related content + +* [Entity groups](entity-groups.md) +* [Entity types](../query/schema-entities/index.md) +* [.create entity_group](create-entity-group.md) +* [.alter-merge entity_group](alter-merge-entity-group.md) +* [.drop entity_group](drop-entity-group.md) +* [.show entity_group and .show entity_groups](show-entity-group.md) diff --git a/data-explorer/kusto/management/alter-merge-entity-group.md b/data-explorer/kusto/management/alter-merge-entity-group.md index 35fe6409c8..fda91801d8 100644 --- a/data-explorer/kusto/management/alter-merge-entity-group.md +++ b/data-explorer/kusto/management/alter-merge-entity-group.md @@ -1,43 +1,76 @@ +--- +title: .alter-merge entity_group +description: Learn how to use the `.alter-merge entity_group` command to change an existing entity group. +ms.reviewer: ziham1531991 +ms.topic: reference +ms.date: 01/20/2025 +--- + # .alter-merge entity_group -Alters and merges an existing entity group with the provided list of entities and stores it inside the database metadata. +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] + +Alters and merges an existing entity group with the provided list of entities and stores it inside the database metadata. For more information, see [Entity groups](entity-groups.md). + +## Permissions + +You must have at least [Database Admin](../access-control/role-based-access-control.md) permissions to run this command. + +## Syntax + +`.alter-merge` `entity_group` *EntityGroupName* `(`[*EntityReference*`,` ...]`)` + +[!INCLUDE [syntax-conventions-note](../includes/syntax-conventions-note.md)] - -**Syntax** +## Parameters -`.alter-merge` `entity_group` *EntityGroupName* `(` EntityReference1 , ... `)` +|Name|Type|Required|Description| +|--|--|--|--| +|*EntityGroupName*| `string` | :heavy_check_mark:|The name of the entity group. | +|*EntityReference*| `string` | :heavy_check_mark:|An entity included in the entity group. | -**Output** - -|Output parameter |Type |Description -|---|---|--- -|Name |String |The name of the entity group. -|Entities |String | An array of the entities defined into the entity group. +## Returns -> [!NOTE] -> * If the entity group doesn't exist, an error is returned. For creating a new entity group, see [.create entity_group](entity-group-create.md) -> * Requires [database admin permission](../management/access-control/role-based-authorization.md) +This command returns a table with the following columns: -**Examples** +|Output parameter |Type |Description| +|---|---|---| +|Name | `string` | The name of the entity group.| +|Entities | `array` | An array which includes one or more entities.| -Running the following command to create a new entity group: +## Examples + +The following example edits the `MyEntityGroup` entity group and adds the entity `cluster('c2').database('d2')` to the entity group. + +First run the following command to create a new entity group with entity `cluster('c1').database('d1')`: ```kusto .create entity_group MyEntityGroup (cluster('c1').database('d1')) ``` +**Output** + |Name|Entities| |---|---| |MyEntityGroup|["cluster('c1').database('d1')"]| - -Then running the following command to edit the existing entity group MyEntityGroup and add the entity `cluster('c2').database('d2')` : +Then run the following command to edit the existing entity group `MyEntityGroup` and add the entity `cluster('c2').database('d2')`: ```kusto .alter-merge entity_group MyEntityGroup (cluster('c2').database('d2')) ``` +**Output** + |Name|Entities| |---|---| |MyEntityGroup|["cluster('c1').database('d1')","cluster('c2').database('d2')"]| +## Related content + +* [Entity groups](entity-groups.md) +* [Entity types](../query/schema-entities/index.md) +* [.create entity_group](create-entity-group.md) +* [.alter entity_group](alter-entity-group.md) +* [.drop entity_group](drop-entity-group.md) +* [.show entity_group and .show entity_groups](show-entity-group.md) diff --git a/data-explorer/kusto/management/create-entity-group.md b/data-explorer/kusto/management/create-entity-group.md index bc32db32d3..8fa8e1b7b1 100644 --- a/data-explorer/kusto/management/create-entity-group.md +++ b/data-explorer/kusto/management/create-entity-group.md @@ -1,28 +1,47 @@ +--- +title: .create entity_group(s) +description: Learn how to use the `.create entity_group` command to create an entity group. +ms.reviewer: ziham1531991 +ms.topic: reference +ms.date: 01/20/2025 +--- + # .create entity_group -Creates a stored entity group which is a reusable [`let` statement](../query/letstatement.md) -entity group with the given name. The entity group definition is persisted with the database metadata. +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] + +Creates a stored entity group with a specific name, which functions like a reusable [`let` statement](../query/letstatement.md). The entity group definition is saved as part of the database metadata. + +## Permissions + +You must have at least [Database Admin](../access-control/role-based-access-control.md) permissions to run this command. + +## Syntax + +`.create` `entity_group` [`ifnotexists`] *EntityGroupName* `(`[*EntityReference*`,` ...]`)` - -**Syntax** +[!INCLUDE [syntax-conventions-note](../includes/syntax-conventions-note.md)] -`.create` `entity_group` [`ifnotexists`] *EntityGroupName* `(` EntityReference1 , ... `)` +## Parameters -**Output** - -|Output parameter |Type |Description -|---|---|--- -|Name |String |The name of the entity group. -|Entities |String | An array of the entities defined into the entity group. +|Name|Type|Required|Description| +|--|--|--|--| +| `ifnotexists` | `string` | | If specified, the entity group only is created if the entity group doesn't exist yet.| +|*EntityGroupName*| `string` | :heavy_check_mark:|The name of the entity group. | +|*EntityReference*| `string` | :heavy_check_mark:|An entity included in the entity group. | -> [!NOTE] -> * If entity group already exists: -> * If `ifnotexists` flag is specified, the command is ignored (no change applied). -> * If `ifnotexists` flag is NOT specified, an error is returned. -> * For altering an existing entity group, see [.alter entity_group](entity-group-alter.md) -> * Requires [database admin permission](../management/access-control/role-based-authorization.md). +## Returns -**Examples** +This command returns a table with the following columns: + +|Output parameter |Type |Description| +|---|---|---| +|Name | `string` | The name of the entity group.| +|Entities | `array` | An array which includes one or more entities. If the entity group already exists, and the `ifnotexists` flag is specified, the command is ignored. Otherwise an error is returned.| + +## Examples + +The following example creates the `MyEntityGroup` entity group with two entities, `cluster('c1').database('d1')` and `cluster('c2').database('d2')`. ```kusto .create entity_group MyEntityGroup (cluster('c1').database('d1'), cluster('c2').database('d2')) @@ -31,3 +50,12 @@ entity group with the given name. The entity group definition is persisted with |Name|Entities| |---|---| |MyEntityGroup|["cluster('c1').database('d1')","cluster('c2').database('d2')"]| + +## Related content + +* [Entity groups](entity-groups.md) +* [Entity types](../query/schema-entities/index.md) +* [.alter entity_group](alter-entity-group.md) +* [.alter-merge entity_group](alter-merge-entity-group.md) +* [.drop entity_group](drop-entity-group.md) +* [.show entity_group and .show entity_groups](show-entity-group.md) \ No newline at end of file diff --git a/data-explorer/kusto/management/drop-entity-group.md b/data-explorer/kusto/management/drop-entity-group.md index 30f539ba76..d6e09e24fa 100644 --- a/data-explorer/kusto/management/drop-entity-group.md +++ b/data-explorer/kusto/management/drop-entity-group.md @@ -1,22 +1,55 @@ +--- +title: .drop entity_group(s) +description: Learn how to use the `.drop entity_group` command to remove an entity group from your database. +ms.reviewer: ziham1531991 +ms.topic: reference +ms.date: 01/20/2025 +--- + # .drop entity_group +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] + Drops an entity group from the database. - -**Syntax** + +## Permissions + +You must have at least [Database Admin](../access-control/role-based-access-control.md) permissions to run this command. + +## Syntax `.drop` `entity_group` *EntityGroupName* -**Output** +[!INCLUDE [syntax-conventions-note](../includes/syntax-conventions-note.md)] + +## Parameters + +|Name|Type|Required|Description| +|--|--|--|--| +|*EntityGroupName*| `string` | |The name of the specific entity group you wish to drop. | + +## Returns This command returns a list of the remaining tables in the database. -|Output parameter |Type |Description -|---|---|--- -|Name |String |The name of the entity group. -|Entities |String | An array of the entities defined into the entity group. +|Output parameter |Type |Description| +|---|---|---| +|Name | `string` | The name of the entity group.| +|Entities | `array` | An array which includes one or more entities.| + +## Examples -**Examples** +The following example drops the `MyEntityGroup` from your selected database. ```kusto .drop entity_group MyEntityGroup ``` + +## Related content + +* [Entity groups](entity-groups.md) +* [Entity types](../query/schema-entities/index.md) +* [.create entity_group](create-entity-group.md) +* [.alter entity_group](alter-entity-group.md) +* [.alter-merge entity_group](alter-merge-entity-group.md) +* [.show entity_group and .show entity_groups](show-entity-group.md) \ No newline at end of file diff --git a/data-explorer/kusto/management/entity-groups.md b/data-explorer/kusto/management/entity-groups.md index f751dee5ef..82cd550535 100644 --- a/data-explorer/kusto/management/entity-groups.md +++ b/data-explorer/kusto/management/entity-groups.md @@ -1,17 +1,33 @@ -# Stored entity group management overview +--- +title: Entity groups +description: Learn how to use Entity groups to store entity groups in the database. +ms.reviewer: ziham1531991 +ms.topic: reference +ms.date: 01/20/2025 +--- -Stored entity groups are named entities that are stored in a database and can be referenced by the [macro-expand query operator](../query/macro-expand-operator.md). -Storing an entity group in the database instead of providing its value in the query text itself makes it easier to manage these objects. +# Entity groups -> [!NOTE] -> A query can only reference entity groups defined in the query text or in the database-in-context. Entity groups in other databases or clusters (directly or indirectly) cannot be referenced. +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] + +Entity groups are named entities that are stored in a database and that the [macro-expand query operator](../query/macro-expand-operator.md) can reference. + +Storing an entity group in the database instead of providing its value in the query text itself makes it easier to manage these objects. -This section describes control commands used for creating, altering and dropping entity groups: +## Create and management entity groups |Function |Description| |---------|-----------| |[.alter entity_group](alter-entity-group.md) |Alters an existing entity group and stores it inside the database metadata | -|[.alter-merge entity_group](alter-merge-entity-group.md) |Alters and merge the value of an existing function | -|[.create entity_group](create-entity-group.md) |Creates a stored entity group| -|[.drop entity_grop](drop-entity-group.md) |Drops an entity group from the database | -|[.show entity_grop and .show entity_groups](show-entity-group.md) |Lists all the stored entity groups, or a specific entity group, in the currently-selected database| +|[.alter-merge entity_group](alter-merge-entity-group.md) |Alters and merges the value of an existing entity group. | +|[.create entity_group](create-entity-group.md) |Creates a stored entity group.| +|[.drop entity_group](drop-entity-group.md) |Drops an entity group from the database. | +|[.show entity_group(s)](show-entity-group.md) |Lists all the stored entity groups, or a specific entity group, in the current database.| + +> [!NOTE] +> A query can only reference entity groups defined in the query text or in the scoped database. Entity groups in other databases or clusters can't be referenced directly or indirectly. + +## Related content + +* [macro-expand operator](../query/macro-expand-operator.md) +* [Entity types](../query/schema-entities/index.md) \ No newline at end of file diff --git a/data-explorer/kusto/management/show-entity-group.md b/data-explorer/kusto/management/show-entity-group.md index 42cc94d405..54a81642e5 100644 --- a/data-explorer/kusto/management/show-entity-group.md +++ b/data-explorer/kusto/management/show-entity-group.md @@ -1,59 +1,80 @@ +--- +title: .show entity_group(s) +description: Learn how to use the `.show entity_group` command to view existing entity groups. +ms.reviewer: ziham1531991 +ms.topic: reference +ms.date: 01/20/2025 +--- + # .show entity_group(s) -Lists all the [stored entity groups](entity-groups.md) in the currently-selected database. -To return only one specific entity group, see [.show entity_group](#show-entity_group). +> [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] + +Lists all the [entity groups](entity-groups.md) in the selected database or lists the details of one specific stored entity group. + +## Permissions + +You must have at least [Database Admin](../access-control/role-based-access-control.md) permissions to run this command. + +## Syntax + +`.show` `entity_group` + +`.show` `entity_group` *EntityGroupName* + +[!INCLUDE [syntax-conventions-note](../includes/syntax-conventions-note.md)] + +## Parameters + +|Name|Type|Required|Description| +|--|--|--|--| +|*EntityGroupName*| `string` | |The name of the specific entity group you wish to view. | + +## Returns + +This command returns a table with the following columns: + +|Output parameter |Type |Description| +|---|---|---| +|Name | `string` | The name of the entity group.| +|Entities | `array` | An array which includes one or more entities. If the entity group doesn't exist, an error is returned.| + +## Examples -## .show entity_groups +### Show entity groups + +The following example returns all the entity groups in the selected database, `eg1` and `eg2`, along with their entities. ```kusto .show entity_groups ``` -Requires [database user permission](../management/access-control/role-based-authorization.md). - -|Output parameter |Type |Description -|---|---|--- -|Name |String |The name of the entity group. -|Entities|String|The value of the entity group as an array that includes the entities. - -**Output example** +**Output** |Name|Entities| |---|---| |eg1|["cluster('c1').database('d1')"]| |eg2|["cluster('c2').database('d2')"]| +### Show an entity group -## .show entity_group +The following example returns the entity group, `eg1` along with its entity, database `d1` in the cluster `c1`. ```kusto -.show entity_group MyEntityGroup1 +.show entity_group eg1 ``` -Lists the details of one specific stored entity group. -For a list of **all** entity groups, see [.show entity_groups](#show-entity_groups). - -**Syntax** - -`.show` `entity_group` *EntityGroupName* - **Output** -|Output parameter |Type |Description -|---|---|--- -|Name |String |The name of the entity group. -|Entities|String|The value of the entity group as an array that includes the entities. - -> [!NOTE] -> * If the entity group does not exist, an error is returned. -> * Requires [database user permission](../management/access-control/role-based-authorization.md). - -**Example** - -```kusto -.show entity_group eg1 -``` - |Name|Entities| |---|---| |eg1|["cluster('c1').database('d1')"]| + +## Related content + +* [Entity groups](entity-groups.md) +* [Entity types](../query/schema-entities/index.md) +* [.alter entity_group](alter-entity-group.md) +* [.alter-merge entity_group](alter-merge-entity-group.md) +* [.create entity_group](create-entity-group.md) +* [.drop entity_group](drop-entity-group.md) \ No newline at end of file diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index f664200a90..6c3a562a6e 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -1,27 +1,28 @@ -# The macro-expand operator +--- +title: macro-expand operator +description: Learn how to use the macro-expand operator to run a subquery on a set of entitites. +ms.reviewer: ziham1531991 +ms.topic: reference +ms.date: 01/20/2025 +--- +# macro-expand operator -The `macro-expand` operator provides concise syntax for applying a subquery -to a set of entities (clusters, databases, or tables) and combining their results. -It may come handy, for example, when one has data spread over multiple -clusters/databases/table (for example, when data is held in the same global region -as its source). The set of entities is called **entity group**; it can be stored for -reuse in a database, or defined ad-hoc in the query text itself. +The `macro-expand` operator simplifies running a subquery on a set of [entities](schema-entities/index.md) entities such as clusters, databases, or tables, and then combining the results into a single output. -> [!NOTE] -> The operation of the `macro-expand` operator can be altered by setting the `best_effort` request property to `true`, using either a `set statement` or through [client request properties](../api/rest/request-properties.md). When this property is set to `true`, the `macro-expand` operator will disregard fuzzy resolution and connectivity failures to execute any of the sub-expressions being “unioned” and yield a warning in the query status results. +The operator is useful when you have data spread across multiple clusters, databases, or tables. For example, when data is held in the same global region as its source, you can use the `macro-expand` operator in a single query across data in distinct locations, instead of running separate queries for each entity and combining the results manually. + +The set of entities you want to query, is called an **entity group**. The entity group can be stored for reuse in a database, or defined directly in your query text. For more information about stored entity groups, see [Entity groups](../management/entity-groups.md). -## Computation model +The `macro-expand` operator runs the subquery separately for each entity in the group and then combines all the results into a single output. + + + -## Syntax +### Syntax -There are several syntactic variants one may use with the `macro-expand` operator: +`macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] *EntityGroup* `as` *EntityIdentifier* `(` *Subquery* `)` -1. Explicit: In this syntax, the operator name (`macro-expand`), identifier, - entity group, and subquery are all explicit in the text of the operator - invocation itself. +`macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] `entity_group` `[` *EntityReference* [`,` ...] `as` *EntityIdentifier* `(` *Subquery* `)` `]` -2. Explicit using let statement: This syntax is similar to the explicit syntax, - with the difference being that the entity group is specified in the query - using a let statement (that is, outside the `macro-expand` operator). +`macro-expand` *EntityIdentifier* `in` *EntityGroupIdentifier* `(` *Subquery* `)` -3. Explicit using stored entity group: This syntax is similar to the syntax - that uses the let statement, only the query itself doesn't define the entity group. - Rather, the group is stored in the database-in-scope as a named entity. + +> [!NOTE] +> The operation of the `macro-expand` operator can be modified by setting the `best_effort` request property to `true`, either by using a `set statement` or through [client request properties](../api/rest/request-properties.md). When this property is set to `true`, the `macro-expand` operator ignores fuzzy resolution and connectivity failures to execute any of the subexpressions being unioned and issues a warning in the query status results. -### Explicit syntax +## Syntax variations -`macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] *EntityGroup* `as` *EntityIdentifier* `(` *Subquery* `)` +There are several syntactic variants that are used with the `macro-expand` operator: -Where: +1. **Explicit:** In this syntax, the `macro-expand` operator, identifier, entity group, and subquery are all explicit in the text of the operator invocation itself. -* *EntityGroup* has the syntax `entity_group` `[` *EntityReference1* [`,` ...] `]`, - and denotes one or more entities that *EntityIdentifier* expands to. +1. **Explicit using let statement:** The entity group is specified in the query using a `let` statement outside the `macro-expand` operator using the syntax: + + `let` *EntityGroupIdentifier* `=` `entity_group` `[` *EntityReference* [`,` ...] `]` + +1. **Explicit using stored entity group:** The query doesn't define the entity group. Instead it uses an entity group stored in the database in scope. + +## Parameters + +| Name | Type | Required | Description | +|--|--|--|--| +| *Kind* | `string` | | Either `inner` or `outer` (default). When `kind` is set to `inner`, the results only include columns common to all the accessed scoped entities. If set to `outer`, the result has all the columns that occur in any of the inputs. Cells that aren't defined by an input row are set to `null`. | +| *IsFuzzy* | | | When set to `true`, it only considers entity sources that currently exist and are accessible. If at least one entity is found, the query runs, and any missing entities generate warnings in the query status results. If no entities are found, the query can't resolve any specified entities and it returns an error. The default is `false`. | +| *EntityGroup* | | :heavy_check_mark: | A set of one or more entities that *EntityIdentifier* expands to when a query is run. The entity group can be a stored entity group or a directly defined group. It denotes one or more entities of the same type that *EntityIdentifier* expands to. | +| *EntityIdentifier* | `string` | :heavy_check_mark: | An identifier that serves as a placeholder for an entity in the *Subquery*, and which is expanded into the actual entity when the query is run. Entities that aren't explicitly scoped in *EntityIdentifier*, are assumed to be part of the current in scope database. Any specific identifiers included in the query override the default assumption. | +|*EntityReference*| `string` | |An entity included in the entity group. One or more *EntityReference* is required if an *EntityGroup* isn't specified. | +| *Subquery* | `string` | :heavy_check_mark: | A single tabular expression that doesn’t take input data directly. It might include references to entities through an *EntityIdentifier*, and use expressions such as let statements, stored functions, or other elements from the database in scope. *Subquery* can also be preceded by one or more `let` statements. It can also reference [specialized scalar functions](#subquery-contextual-functions).| > [!NOTE] -> A query can only reference entity groups defined in the query text or in the database-in-context. Entity groups in other databases or clusters (directly or indirectly) cannot be referenced. +> A query can only reference entity groups defined in the query text or in the current database. Entity groups in other databases or clusters can't be referenced directly or indirectly. -* *EntityIdentifier* is an identifier whose appearance in *Subquery* - gets expanded into each of the entities in the group. +### Subquery contextual functions -* *Subquery* is a tabular expression (optionally preceded by one or more let - statements) that takes no source and may reference *EntityIdentifier*. - All entity references other than *EntityIndentifier* behave "normally"; they - reference other query let statements, stored functions, etc. from the - database-in-scope. +The `macro-expand` subquery can reference two specialized scalar functions as if they're part of the entity being referenced: -* *Kind* can take one of two values: - - `inner` - The result has the subset of columns that are common to all of the accessed scoped entities. - - `outer` - The result has all the columns that occur in any of the inputs. Cells that weren't defined by an input row are set to null. This is the default. +1. `$current_database` - Returns the database name of the entity reference. +1. `$current_cluster_endpoint` - Returns the URL of the cluster of the entity reference. + +## Examples -* *IsFuzzy* (is set to `true`) enables fuzzy resolution of macro-expand scoped entities. - Fuzzy applies to the set of union sources. It means that while analyzing the query and preparing for execution, - the set of scope sources is reduced to the set of entities that exist and are accessible at the time. - If at least one such entity was found, any resolution failure will yield a warning in the query status results - (one for each missing reference), but will not prevent the query execution; - if no resolutions were successful - the query will return an error. The default is `false`. +### Calculate errors -For example, the query below calculates the number of errors produced by each SKU. +The following example uses an entity group to calculate the number of errors produced by each Stock Keeping Unit (SKU). It defines an `entity_group`, `X` that includes databases named `Kuskus` in two clusters. The query then performs a subquery where it filters for error logs, counts the errors by `Source`, and performs an `inner` join on `Source` with the `DimCluster` table to get the `SKU` for each source. Finally it sums the error counts by `SKU`. ```kusto macro-expand entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')] as X @@ -94,7 +97,15 @@ macro-expand entity_group [cluster('C1').database('Kuskus'), cluster('C2').datab | summarize Sum=sum(Count) by SKU ``` -Here's another way to write the same query without using `macro-expand`: +1. The `summarize` operator works on the combined results of all expanded subqueries while the `macro-expand` operator is only applied to the subquery between the parentheses so it's clear which scope is expanded. + +1. The `join` operator is executed separately for each entity in the `entity_group`. In the example, the `join` is performed between two tables in the same entity, denoted by `X`. This means there was no cross-cluster `join`. + +1. While not shown in this example, the subquery can include nested `macro-expand` operators. This means you can use the `macro-expand` operator within another `macro-expand` operator. However, the identifier for the inner `macro-expand` must be different from the identifier for the outer `macro-expand` to clearly distinguish between the scope and references of each one. + +1. If `IsFuzzy` is set to `true`, the query warns about missing SKUs but still runs. If no SKUs are found, the query fails. + +To write the same query without using `macro-expand`, it might look as follows: ```kusto union @@ -113,35 +124,25 @@ union | summarize Sum=sum(Count) by SKU ``` -In this example: - -1. The `entity_group` is a collection of databases from different clusters. - -1. The identifier `X` is used to scope entity references. Entities that are - are not scoped by this identifier are taken to be in the database-in-scope - scope (if they are not overridden as query-provided identifiers.) - -1. The `summarize` operator operates on the `union` of all expanded subqueries. - As the `expand` operator get applied only to the subquery between the `(` and `)` - characters, it's clear which scope is expanded and which is not. - -1. The `join` operator above is calculated two times, one for each entity in the - `entity_group`. Also note that `join` here is being done between two tables - of the same entity (denoted by `X`) &emdash; there's no cross-cluster `join` here. +### Calculate SKU errors using `let` statement -1. Not shown in this example, the subquery part can include additional uses of - the `macro-expand` operator (nested expansions). Note that the identifier of the inner - `macro-expand` cannot equal that of the outer, providing a clear way to distinguish the - scope. +The following example uses a `let` statement to define an entity group named `UberKuskus` that includes the `Kuskus` database from both `C1` and `C2` clusters. It then uses that to perform the same query in the previous example to calculate the number of errors produced by each SKU. It uses the `macro-expand` operator to reference the `UberKuskus` entity group (alias `X`). -### Special Contextual functions - -The `macro-expand` subquery can reference two specialized scalar functions as-if they are part of the entity being referenced: +```kusto +let UberKuskus = entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')]; +macro-expand UberKuskus as X +( + X.KustoLogs + | where Level == 'Error' + | summarize Count=count() by Source + | join kind=inner (X.DimCluster | project SKU, Source) on Source +) +| summarize Sum=sum(Count) by SKU +``` -1. `$current_database` - Returns the database name of the entity reference. -2. `$current_cluster_endpoint` - Returns the URL of the cluster of the entity reference. +### Extend table with contextual scalar functions -Examples: +The following query runs a subquery on the `Admins` table from each entity in the `MyEntityGroup`. It uses `$current_database` and `$current_cluster_endpoint` to extend the table to include the current database and current cluster for each row. It then summarizes the results by counting the number of rows for each combination of `current_cluster` and `current_database`. ```kusto macro-expand MyEntityGroup as X @@ -152,6 +153,10 @@ macro-expand MyEntityGroup as X | summarize count() by current_cluster, current_database ``` +### Nested macro-expand query + +The following query runs a nested subquery with an outer entity group `MyEntityGroup_Outer` (alias `X`) and an inner entity group `MyEntityGroup_Inner` (alias `Y`). It joins the `Admins` table from each entity in both the outer and inner entity groups. The query filters for logs from the last hour, extends the tables to include the current database and cluster for each row using `$current_database` and `$current_cluster_endpoint`, and performs a `join` on the `Source` column. The prefixes `lhs` and `rhs` denote `X` and `Y` entity groups respectively. Finally, it summarizes the results by counting the number of rows for each combination of `lhs_cluster`, `lhs_database`, `rhs_cluster`, and `rhs_database`. + ```kusto macro-expand MyEntityGroup_Outer as X ( @@ -170,30 +175,11 @@ macro-expand MyEntityGroup_Outer as X | summarize count() by lhs_cluster, lhs_database, rhs_cluster, rhs_database ``` -### Explicit using let statement - -`macro-expand` *EntityIdentifier* `in` *EntityGroupIdentifier* `(` *Subquery* `)` - -Here, *EntityGroupIdentifier* is simply an identifier which has been defined by -the query using a let statement: - -`let` *EntityGroupIdentifier* `=` `entity_group` `[` *EntityReference1* [`,` ...] `]` - -For example, the previous example could be re-written as: - -```kusto -let UberKuskus = entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')]; -macro-expand UberKuskus as X -( - X.KustoLogs - | where Level == 'Error' - | summarize Count=count() by Source - | join kind=inner (X.DimCluster | project SKU, Source) on Source -) -| summarize Sum=sum(Count) by SKU -``` - -### Explicit using stored entity group +## Related content -The syntax is exactly the same as for using the let statement. -See here [Stored Entity Groups Management](../management/entity-groups.md) +* [Entity groups](../management/entity-groups.md) +* [join operator](join-operator.md) +* [union operator](union-operator.md) +* [.show entity_group(s)](show-entity-group.md) +* [current_database()](current-database-function.md) +* [current_cluster_endpoint()](current-cluster-endpoint-function.md) From 12df44507e2a03ca15a6f12e2c1a131bc3d1ae64 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Mon, 20 Jan 2025 17:33:45 +0200 Subject: [PATCH 18/37] edits --- data-explorer/kusto/management/create-entity-group.md | 2 +- data-explorer/kusto/management/toc.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/data-explorer/kusto/management/create-entity-group.md b/data-explorer/kusto/management/create-entity-group.md index 8fa8e1b7b1..6c798b4f13 100644 --- a/data-explorer/kusto/management/create-entity-group.md +++ b/data-explorer/kusto/management/create-entity-group.md @@ -10,7 +10,7 @@ ms.date: 01/20/2025 > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] -Creates a stored entity group with a specific name, which functions like a reusable [`let` statement](../query/letstatement.md). The entity group definition is saved as part of the database metadata. +Creates a stored entity group with a specific name, which functions like a reusable [`let` statement](../query/let-statement.md). The entity group definition is saved as part of the database metadata. ## Permissions diff --git a/data-explorer/kusto/management/toc.yml b/data-explorer/kusto/management/toc.yml index 67b5ab32b7..ff57170a62 100644 --- a/data-explorer/kusto/management/toc.yml +++ b/data-explorer/kusto/management/toc.yml @@ -255,7 +255,7 @@ items: - name: .alter-merge table column-docstrings command displayName: .alter-merge column-docstrings href: alter-merge-table-column.md - - name: Entity Group + - name: Entity Group items: - name: Entity Group href: entity-groups.md From aef3d2e54785e121434cf575ed9a9bc0ba57778f Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Mon, 20 Jan 2025 17:44:04 +0200 Subject: [PATCH 19/37] edit --- data-explorer/kusto/query/macro-expand-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 6c3a562a6e..3e709fbea5 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -180,6 +180,6 @@ macro-expand MyEntityGroup_Outer as X * [Entity groups](../management/entity-groups.md) * [join operator](join-operator.md) * [union operator](union-operator.md) -* [.show entity_group(s)](show-entity-group.md) +* [.show entity_group(s)](../management/show-entity-group.md) * [current_database()](current-database-function.md) * [current_cluster_endpoint()](current-cluster-endpoint-function.md) From 32ea6ec33b3b7504f7e08f03355179a7ad5c5e88 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Mon, 20 Jan 2025 18:24:46 +0200 Subject: [PATCH 20/37] edit --- .../kusto/management/show-entity-group.md | 2 +- .../kusto/query/macro-expand-operator.md | 29 ++++++++----------- 2 files changed, 13 insertions(+), 18 deletions(-) diff --git a/data-explorer/kusto/management/show-entity-group.md b/data-explorer/kusto/management/show-entity-group.md index 54a81642e5..eb69aa9b1a 100644 --- a/data-explorer/kusto/management/show-entity-group.md +++ b/data-explorer/kusto/management/show-entity-group.md @@ -58,7 +58,7 @@ The following example returns all the entity groups in the selected database, ` ### Show an entity group -The following example returns the entity group, `eg1` along with its entity, database `d1` in the cluster `c1`. +The following example returns the entity group, `eg1` along with its entity, `cluster('c1').database('d1')`. ```kusto .show entity_group eg1 diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 3e709fbea5..62f071407d 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -7,14 +7,13 @@ ms.date: 01/20/2025 --- # macro-expand operator -The `macro-expand` operator simplifies running a subquery on a set of [entities](schema-entities/index.md) entities such as clusters, databases, or tables, and then combining the results into a single output. +The `macro-expand` operator simplifies running a subquery on a set of [entities](schema-entities/index.md) such as clusters, databases, or tables, and then combining the results into a single output. The operator is useful when you have data spread across multiple clusters, databases, or tables. For example, when data is held in the same global region as its source, you can use the `macro-expand` operator in a single query across data in distinct locations, instead of running separate queries for each entity and combining the results manually. The set of entities you want to query, is called an **entity group**. The entity group can be stored for reuse in a database, or defined directly in your query text. For more information about stored entity groups, see [Entity groups](../management/entity-groups.md). -The `macro-expand` operator runs the subquery separately for each entity in the group and then combines all the results into a single output. - +The `macro-expand` operator runs the subquery separately for each entity in the group and then combines all the results into a single output. The subquery can include nested `macro-expand` operators. This means you can use the `macro-expand` operator within another `macro-expand` operator. However, the identifier for the inner `macro-expand` must be different from the identifier for the outer `macro-expand` to clearly distinguish between the scope and references of each one. - - -### Syntax +## Syntax `macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] *EntityGroup* `as` *EntityIdentifier* `(` *Subquery* `)` @@ -44,15 +25,14 @@ as many times as there are individual entities in the entity group. `macro-expand` *EntityIdentifier* `in` *EntityGroupIdentifier* `(` *Subquery* `)` - > [!NOTE] > The operation of the `macro-expand` operator can be modified by setting the `best_effort` request property to `true`, either by using a `set statement` or through [client request properties](../api/rest/request-properties.md). When this property is set to `true`, the `macro-expand` operator ignores fuzzy resolution and connectivity failures to execute any of the subexpressions being unioned and issues a warning in the query status results. -## Syntax variations +### Syntax variations There are several syntactic variants that are used with the `macro-expand` operator: -1. **Explicit:** The `macro-expand` operator, identifier, entity group, and subquery are all explicitly defined in the text of the operator invocation itself. +1. **Explicit:** All elements are explicitly defined in the text of the operator invocation itself. 1. **Explicit using let statement:** The entity group is specified in the query using a `let` statement outside the `macro-expand` operator using the syntax: @@ -60,16 +40,18 @@ There are several syntactic variants that are used with the `macro-expand` opera 1. **Explicit using stored entity group:** The query uses an entity group stored in the database in scope rather than defined in the query. +To see examples of how the variants are used, see [Examples](#examples). + ## Parameters | Name | Type | Required | Description | |--|--|--|--| | *Kind* | `string` | | Either `inner` or `outer` (default). When `kind` is set to `inner`, the results only include columns common to all the accessed scoped entities. If set to `outer`, the result includes all the columns that occur in any of the inputs. Cells that aren't defined by an input row are set to `null`. | | *IsFuzzy* | | | When set to `true`, it only considers entity sources that currently exist and are accessible. If at least one entity is found, the query runs, and any missing entities generate warnings in the query status results. If no entities are found, the query can't resolve any specified entities and returns an error. The default is `false`. | -| *EntityGroup* | | :heavy_check_mark: | A set of one or more entities that *EntityIdentifier* expands to when a query is run. The entity group can be a stored entity group or a directly defined group. It denotes one or more entities of the same type that *EntityIdentifier* expands to. | +| *EntityGroup* | | :heavy_check_mark: | A set of one or more entities that *EntityIdentifier* expands to when a query is run. The entity group can be a stored entity group or a defined group. It denotes one or more entities of the same type that *EntityIdentifier* expands to. | | *EntityIdentifier* | `string` | :heavy_check_mark: | An identifier that serves as a placeholder for an entity in the subquery, and which is expanded into the actual entity when the query is run. Entities that aren't explicitly scoped in *EntityIdentifier* are assumed to be part of the current in scope database. Any specific identifiers included in the query override the default assumption. | |*EntityReference*| `string` | |An entity included in the entity group. One or more *EntityReference* is required if an *EntityGroup* isn't specified. | -| *Subquery* | `string` | :heavy_check_mark: | A single tabular expression that doesn’t take input data directly. It might include references to entities through an *EntityIdentifier*, and use expressions such as let statements, stored functions, or other elements from the database in scope. *Subquery* can also be preceded by one or more `let` statements. It can also reference [specialized scalar functions](#subquery-contextual-functions).| +| *Subquery* | `string` | :heavy_check_mark: | A single tabular expression that doesn’t take input data, directly. It might include references to entities through an *EntityIdentifier*, and use expressions such as let statements, stored functions, or other elements from the database in scope. *Subquery* can also be preceded by one or more `let` statements. It can also reference [Subquery contextual functions](#subquery-contextual-functions).| > [!NOTE] > A query can only reference entity groups defined in the query text or in the current database. Entity groups in other databases or clusters can't be referenced directly or indirectly. From e2cf6e4007a6581b69c09e329d57e5ff3a051bd8 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Mon, 27 Jan 2025 08:54:35 +0200 Subject: [PATCH 23/37] edits --- .../kusto/management/alter-entity-group.md | 8 +++--- .../management/alter-merge-entity-group.md | 4 +-- .../kusto/management/create-entity-group.md | 6 ++--- .../kusto/management/drop-entity-group.md | 6 ++--- .../kusto/management/entity-groups.md | 10 +++---- .../kusto/management/show-entity-group.md | 2 +- .../kusto/query/macro-expand-operator.md | 26 +++++++++---------- 7 files changed, 31 insertions(+), 31 deletions(-) diff --git a/data-explorer/kusto/management/alter-entity-group.md b/data-explorer/kusto/management/alter-entity-group.md index 0d30ab87ee..7196c28549 100644 --- a/data-explorer/kusto/management/alter-entity-group.md +++ b/data-explorer/kusto/management/alter-entity-group.md @@ -3,7 +3,7 @@ title: .alter entity_group description: Learn how to use the `.alter entity_group` command to change an existing entity group. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/21/2025 +ms.date: 01/26/2025 --- @@ -19,7 +19,7 @@ You must have at least [Database Admin](../access-control/role-based-access-cont ## Syntax -`.alter` `entity_group` [`ifnotexists`] *EntityGroupName* `(`[*EntityReference*`,` ...]`)` +`.alter` `entity_group` [`ifnotexists`] *EntityGroupName* `(`*EntityReference* [`,` ...]`)` [!INCLUDE [syntax-conventions-note](../includes/syntax-conventions-note.md)] @@ -27,7 +27,7 @@ You must have at least [Database Admin](../access-control/role-based-access-cont |Name|Type|Required|Description| |--|--|--|--| -| `ifnotexists` | `string` | | If specified, the entity group only is created if the entity group doesn't exist yet.| +| `ifnotexists` | `string` | | If specified, the entity group is only created if the entity group doesn't exist yet.| |*EntityGroupName*| `string` | :heavy_check_mark:|The name of the entity group. | |*EntityReference*| `string` | :heavy_check_mark:|An entity included in the entity group. | @@ -42,7 +42,7 @@ This command returns a table with the following columns: ## Examples -The following example alters `MyEntityGroup` to include the entity `cluster('c1').database('d1')`. +The following example alters `MyEntityGroup` to include the entity, `cluster('c1').database('d1')`. ```kusto .alter entity_group MyEntityGroup (cluster('c1').database('d1')) diff --git a/data-explorer/kusto/management/alter-merge-entity-group.md b/data-explorer/kusto/management/alter-merge-entity-group.md index 1a73b4c11c..de4f0446af 100644 --- a/data-explorer/kusto/management/alter-merge-entity-group.md +++ b/data-explorer/kusto/management/alter-merge-entity-group.md @@ -3,7 +3,7 @@ title: .alter-merge entity_group description: Learn how to use the `.alter-merge entity_group` command to change an existing entity group. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/21/2025 +ms.date: 01/26/2025 --- # .alter-merge entity_group @@ -18,7 +18,7 @@ You must have at least [Database Admin](../access-control/role-based-access-cont ## Syntax -`.alter-merge` `entity_group` *EntityGroupName* `(`[*EntityReference*`,` ...]`)` +`.alter-merge` `entity_group` *EntityGroupName* `(`*EntityReference* [`,` ...]`)` [!INCLUDE [syntax-conventions-note](../includes/syntax-conventions-note.md)] diff --git a/data-explorer/kusto/management/create-entity-group.md b/data-explorer/kusto/management/create-entity-group.md index f60984544f..9e3dc38b40 100644 --- a/data-explorer/kusto/management/create-entity-group.md +++ b/data-explorer/kusto/management/create-entity-group.md @@ -3,7 +3,7 @@ title: .create entity_group(s) description: Learn how to use the `.create entity_group` command to create an entity group. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/21/2025 +ms.date: 01/26/2025 --- # .create entity_group @@ -26,7 +26,7 @@ You must have at least [Database Admin](../access-control/role-based-access-cont |Name|Type|Required|Description| |--|--|--|--| -| `ifnotexists` | `string` | | If specified, the entity group only is created if the entity group doesn't exist yet.| +| `ifnotexists` | `string` | | If specified, the entity group is only created if the entity group doesn't exist yet.| |*EntityGroupName*| `string` | :heavy_check_mark:|The name of the entity group. | |*EntityReference*| `string` | :heavy_check_mark:|An entity included in the entity group. | @@ -37,7 +37,7 @@ This command returns a table with the following columns: |Output parameter |Type |Description| |---|---|---| |Name | `string` | The name of the entity group.| -|Entities | `array` | An array which includes one or more entities. If the entity group already exists, and the `ifnotexists` flag is specified, the command is ignored. Otherwise an error is returned.| +|Entities | `array` | An array which includes one or more entities. If the entity group already exists, and the `ifnotexists` flag is specified, the command is ignored. Otherwise, an error is returned.| ## Examples diff --git a/data-explorer/kusto/management/drop-entity-group.md b/data-explorer/kusto/management/drop-entity-group.md index ab3160baec..f675710469 100644 --- a/data-explorer/kusto/management/drop-entity-group.md +++ b/data-explorer/kusto/management/drop-entity-group.md @@ -3,7 +3,7 @@ title: .drop entity_group description: Learn how to use the `.drop entity_group` command to remove an entity group from your database. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/21/2025 +ms.date: 01/26/2025 --- # .drop entity_group @@ -26,7 +26,7 @@ You must have at least [Database Admin](../access-control/role-based-access-cont |Name|Type|Required|Description| |--|--|--|--| -|*EntityGroupName*| `string` | |The name of the specific entity group you wish to drop. | +|*EntityGroupName*| `string` | |The name of the specific entity group you want to drop. | ## Returns @@ -35,7 +35,7 @@ This command returns a list of the remaining tables in the database. |Output parameter |Type |Description| |---|---|---| |Name | `string` | The name of the entity group.| -|Entities | `array` | An array which includes one or more entities.| +|Entities | `array` | An array with one or more entities.| ## Examples diff --git a/data-explorer/kusto/management/entity-groups.md b/data-explorer/kusto/management/entity-groups.md index 0a1170432b..e80fc1dabf 100644 --- a/data-explorer/kusto/management/entity-groups.md +++ b/data-explorer/kusto/management/entity-groups.md @@ -3,30 +3,30 @@ title: Entity groups description: Learn how to use Entity groups to store entity groups in the database. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/21/2025 +ms.date: 01/26/2025 --- # Entity groups > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] -Entity groups are named entities that are stored in a database and that the [macro-expand query operator](../query/macro-expand-operator.md) can reference. +Entity groups are named entities stored in a database that the [macro-expand query operator](../query/macro-expand-operator.md) can reference. -Storing an entity group in the database instead of providing its value in the query text itself makes it easier to manage these objects. +Storing an entity group in the database, instead of providing its value in the query text itself, makes it easier to manage these objects. ## Management commands |Function |Description| |---------|-----------| -|[.alter entity_group](alter-entity-group.md) |Alters an existing entity group and stores it inside the database metadata | +|[.alter entity_group](alter-entity-group.md) |Alters an existing entity group and stores it inside the database metadata. | |[.alter-merge entity_group](alter-merge-entity-group.md) |Alters and merges the value of an existing entity group. | |[.create entity_group](create-entity-group.md) |Creates a stored entity group.| |[.drop entity_group](drop-entity-group.md) |Drops an entity group from the database. | |[.show entity_group(s)](show-entity-group.md) |Lists all the stored entity groups, or a specific entity group, in the current database.| > [!NOTE] -> A query can only reference entity groups defined in the query text or in the scoped database. Entity groups in other databases or clusters can't be referenced directly or indirectly. +> A query can only reference entity groups defined in the query text or in the scoped database. Out-of-scope entity groups that aren't within the query or database can’t be directly or indirectly referenced. ## Related content diff --git a/data-explorer/kusto/management/show-entity-group.md b/data-explorer/kusto/management/show-entity-group.md index 4d4909f890..06c68fa8a4 100644 --- a/data-explorer/kusto/management/show-entity-group.md +++ b/data-explorer/kusto/management/show-entity-group.md @@ -3,7 +3,7 @@ title: .show entity_group(s) description: Learn how to use the `.show entity_group` command to view existing entity groups. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/21/2025 +ms.date: 01/26/2025 --- # .show entity_group(s) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 26829225c9..61074417af 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -3,30 +3,30 @@ title: macro-expand operator description: Learn how to use the macro-expand operator to run a subquery on a set of entities. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/20/2025 +ms.date: 01/27/2025 --- # macro-expand operator > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] -The `macro-expand` operator simplifies running a subquery on a set of [entities](schema-entities/index.md) such as clusters, databases, or tables, and then combining the results into a single output. +The `macro-expand` operator simplifies running a subquery on a set of [entities](schema-entities/index.md), such as clusters, databases, or tables, and then combining the results into a single output. The operator is useful when you have data spread across multiple clusters, databases, or tables. For example, when data is held in the same global region as its source, you can use the `macro-expand` operator in a single query across data in distinct locations, instead of running separate queries for each entity and combining the results manually. -The set of entities you want to query, is called an **entity group**. The entity group can be stored for reuse in a database, or defined directly in your query text. For more information about stored entity groups, see [Entity groups](../management/entity-groups.md). +The set of entities you want to query is called an **entity group**. The entity group can be stored for reuse in a database or defined directly in your query text. For more information about stored entity groups, see [Entity groups](../management/entity-groups.md). -The `macro-expand` operator runs the subquery separately for each entity in the group and then combines all the results into a single output. The subquery can include nested `macro-expand` operators. This means you can use the `macro-expand` operator within another `macro-expand` operator. However, the identifier for the inner `macro-expand` must be different from the identifier for the outer `macro-expand` to clearly distinguish between the scope and references of each one. +The `macro-expand` operator runs the subquery separately for each entity in the group and then combines all the results into a single output. The subquery can include nested `macro-expand` operators. However, the identifier for the inner `macro-expand` must be different from the identifier for the outer `macro-expand` to clearly distinguish between the scope and references of each one. ## Syntax `macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] *EntityGroup* `as` *EntityIdentifier* `(` *Subquery* `)` -`macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] `entity_group` `[` *EntityReference* [`,` ...] `as` *EntityIdentifier* `(` *Subquery* `)` `]` +`macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] `entity_group` `[`*EntityReference* [`,` ...] `as` *EntityIdentifier* `(`*Subquery*`)``]` -`macro-expand` *EntityIdentifier* `in` *EntityGroupIdentifier* `(` *Subquery* `)` +`macro-expand` *EntityIdentifier* `in` *EntityGroupIdentifier* `(`*Subquery*`)` > [!NOTE] -> The operation of the `macro-expand` operator can be modified by setting the `best_effort` request property to `true`, either by using a `set statement` or through [client request properties](../api/rest/request-properties.md). When this property is set to `true`, the `macro-expand` operator ignores fuzzy resolution and connectivity failures to execute any of the subexpressions being unioned and issues a warning in the query status results. +> The operation of the `macro-expand` operator can be modified by setting the `best_effort` request property to `true`, either by using a `set statement` or through [client request properties](../api/rest/request-properties.md). When this property is set to `true`, the `macro-expand` operator ignores fuzzy resolution and connectivity failures, to execute any of the subexpressions being unioned and issues a warning in the query status results. ### Syntax variations @@ -36,7 +36,7 @@ There are several syntactic variants that are used with the `macro-expand` opera 1. **Explicit using let statement:** The entity group is specified in the query using a `let` statement outside the `macro-expand` operator using the syntax: - `let` *EntityGroupIdentifier* `=` `entity_group` `[` *EntityReference* [`,` ...] `]` + `let` *EntityGroupIdentifier* `=` `entity_group` `[`*EntityReference* [`,` ...]`]` 1. **Explicit using stored entity group:** The query uses an entity group stored in the database in scope rather than defined in the query. @@ -46,12 +46,12 @@ To see examples of how the variants are used, see [Examples](#examples). | Name | Type | Required | Description | |--|--|--|--| -| *Kind* | `string` | | Either `inner` or `outer` (default). When `kind` is set to `inner`, the results only include columns common to all the accessed scoped entities. If set to `outer`, the result includes all the columns that occur in any of the inputs. Cells that aren't defined by an input row are set to `null`. | +| *Kind* | `string` | | Either `inner` or `outer` (default). When `kind` is set to `inner`, the results only include columns common to all the accessed scoped entities. If set to `outer`, the result includes all the columns that occur in any of the inputs. Cells not defined by an input row are set to `null`. | | *IsFuzzy* | | | When set to `true`, it only considers entity sources that currently exist and are accessible. If at least one entity is found, the query runs, and any missing entities generate warnings in the query status results. If no entities are found, the query can't resolve any specified entities and returns an error. The default is `false`. | -| *EntityGroup* | | :heavy_check_mark: | A set of one or more entities that *EntityIdentifier* expands to when a query is run. The entity group can be a stored entity group or a defined group. It denotes one or more entities of the same type that *EntityIdentifier* expands to. | +| *EntityGroup* | | :heavy_check_mark: | A set of one or more entities that *EntityIdentifier* expands into when a query is run. The entity group can be a stored entity group or a defined group. It denotes one or more entities of the same type that *EntityIdentifier* expands to. | | *EntityIdentifier* | `string` | :heavy_check_mark: | An identifier that serves as a placeholder for an entity in the subquery, and which is expanded into the actual entity when the query is run. Entities that aren't explicitly scoped in *EntityIdentifier* are assumed to be part of the current in scope database. Any specific identifiers included in the query override the default assumption. | |*EntityReference*| `string` | |An entity included in the entity group. One or more *EntityReference* is required if an *EntityGroup* isn't specified. | -| *Subquery* | `string` | :heavy_check_mark: | A single tabular expression that doesn’t take input data, directly. It might include references to entities through an *EntityIdentifier*, and use expressions such as let statements, stored functions, or other elements from the database in scope. *Subquery* can also be preceded by one or more `let` statements. It can also reference [Subquery contextual functions](#subquery-contextual-functions).| +| *Subquery* | `string` | :heavy_check_mark: | A single tabular expression that doesn’t take input data directly. It might include references to entities through an *EntityIdentifier*, and use expressions such as let statements, stored functions, or other elements from the database in scope. *Subquery* can be preceded by one or more `let` statements. It can also reference [Subquery contextual functions](#subquery-contextual-functions).| > [!NOTE] > A query can only reference entity groups defined in the query text or in the current database. Entity groups in other databases or clusters can't be referenced directly or indirectly. @@ -67,7 +67,7 @@ The `macro-expand` subquery can reference two specialized scalar functions as if ### Calculate errors -The following example uses an entity group to calculate the number of errors produced by each Stock Keeping Unit (SKU). It defines an `entity_group`, `X` that includes databases named `Kuskus` in two clusters. The query then performs a subquery where it filters for error logs, counts the errors by `Source`, and performs an `inner` join on `Source` with the `DimCluster` table to get the `SKU` for each source. Finally, it sums the error counts by `SKU`. +The following example uses an entity group to calculate the number of errors produced by each Stock Keeping Unit (SKU). It defines an `entity_group`, `X`, that includes databases named `Kuskus` in two clusters. The query then performs a subquery where it filters for error logs, counts the errors by `Source`, and performs an `inner` join on `Source` with the `DimCluster` table to get the `SKU` for each source. Finally, it sums the error counts by `SKU`. ```kusto macro-expand entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')] as X @@ -105,7 +105,7 @@ union ### Calculate SKU errors using `let` statement -The following example uses a `let` statement to define an entity group named `UberKuskus` that includes the `Kuskus` database from both `C1` and `C2` clusters. It then uses that to perform the same query in the previous example to calculate the number of errors produced by each SKU. It uses the `macro-expand` operator to reference the `UberKuskus` entity group (alias `X`). +The following example uses a `let` statement to define an entity group named `UberKuskus` that includes the `Kuskus` database from both `C1` and `C2` clusters. It then uses that to perform the same query in the previous example to calculate the number of errors produced by each SKU. It uses the `macro-expand` operator to reference the `UberKuskus` entity group (alias `X`). ```kusto let UberKuskus = entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')]; From 39aa76ad52b7932f3160e93dd6d750a65f89ba8a Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Mon, 27 Jan 2025 14:28:20 +0200 Subject: [PATCH 24/37] edits --- data-explorer/kusto/management/toc.yml | 5 +++++ data-explorer/kusto/query/macro-expand-operator.md | 8 ++++---- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/data-explorer/kusto/management/toc.yml b/data-explorer/kusto/management/toc.yml index b9e77ca139..9e27001792 100644 --- a/data-explorer/kusto/management/toc.yml +++ b/data-explorer/kusto/management/toc.yml @@ -260,14 +260,19 @@ items: - name: Entity Group href: entity-groups.md - name: .create entity_group command + displayName: .create entity_group href: create-entity-group.md - name: .alter entity_group command + displayName: .alter entity_group href: alter-entity-group.md - name: .alter-merge entity_group command + displayName: .alter-merge entity_group href: alter-merge-entity-group.md - name: .drop entity_group command + displayName: .drop entity_group href: drop-entity-group.md - name: .show entity_group command + displayName: .show entity_group href: show-entity-group.md - name: Functions items: diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 61074417af..dd68170c90 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -67,7 +67,7 @@ The `macro-expand` subquery can reference two specialized scalar functions as if ### Calculate errors -The following example uses an entity group to calculate the number of errors produced by each Stock Keeping Unit (SKU). It defines an `entity_group`, `X`, that includes databases named `Kuskus` in two clusters. The query then performs a subquery where it filters for error logs, counts the errors by `Source`, and performs an `inner` join on `Source` with the `DimCluster` table to get the `SKU` for each source. Finally, it sums the error counts by `SKU`. +The following example uses an entity group to calculate the number of errors produced by each Stock Keeping Unit (SKU). It defines an `entity_group`, `X`, that includes databases named `Kuskus` in two clusters. The query then performs a subquery to filter for error logs and counts the errors by `Source`. Next it performs an `inner` join on `Source` with the `DimCluster` table to get the `SKU` for each source. Finally, it sums the error counts by `SKU`. ```kusto macro-expand entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')] as X @@ -105,7 +105,7 @@ union ### Calculate SKU errors using `let` statement -The following example uses a `let` statement to define an entity group named `UberKuskus` that includes the `Kuskus` database from both `C1` and `C2` clusters. It then uses that to perform the same query in the previous example to calculate the number of errors produced by each SKU. It uses the `macro-expand` operator to reference the `UberKuskus` entity group (alias `X`). +The following example uses a `let` statement to define an entity group named `UberKuskus` which includes the `Kuskus` database from both `C1` and `C2` clusters. This entity group is then used to perform the same query in the [previous example](#calculate-errors) to calculate the number of errors produced by each SKU. The `macro-expand` operator is used to reference the `UberKuskus` entity group (alias `X`). ```kusto let UberKuskus = entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')]; @@ -121,7 +121,7 @@ macro-expand UberKuskus as X ### Extend table with contextual scalar functions -The following query runs a subquery on the `Admins` table from each entity in the `MyEntityGroup`. It uses `$current_database` and `$current_cluster_endpoint` to extend the table to include the current database and current cluster for each row. It then summarizes the results by counting the number of rows for each combination of `current_cluster` and `current_database`. +The following query runs a subquery on the `Admins` table from each entity using [a stored entity group](../management/create-entity-group.md#examples), `MyEntityGroup`. It uses `$current_database` and `$current_cluster_endpoint` to extend the table, adding the current database and current cluster for each row. Then, it summarizes the results by counting the number of rows for each combination of `current_cluster` and `current_database`. ```kusto macro-expand MyEntityGroup as X @@ -134,7 +134,7 @@ macro-expand MyEntityGroup as X ### Nested macro-expand query -The following query runs a nested subquery with an outer entity group `MyEntityGroup_Outer` (alias `X`) and an inner entity group `MyEntityGroup_Inner` (alias `Y`). It joins the `Admins` table from each entity in both the outer and inner entity groups. The query filters for logs from the last hour, extends the tables to include the current database and cluster for each row using `$current_database` and `$current_cluster_endpoint`, and performs a `join` on the `Source` column. The prefixes `lhs` and `rhs` denote `X` and `Y` entity groups respectively. Finally, it summarizes the results by counting the number of rows for each combination of `lhs_cluster`, `lhs_database`, `rhs_cluster`, and `rhs_database`. +The following query runs a nested subquery with an outer entity group `MyEntityGroup_Outer` (alias `X`) and an inner entity group `MyEntityGroup_Inner` (alias `Y`). It joins the `Admins` table from each entity in both the outer (`X`) and inner (`Y`) entity groups. The query filters for logs from the last hour. Then it extends the tables to include the current database and cluster for each row using `$current_database` and `$current_cluster_endpoint`. The query performs a `join` on the `Source` column to combine inner and outer entity groups. The prefixes `lhs` (left-hand side) and `rhs` (right-hand side) denote `X` and `Y` entity groups respectively. Finally, it summarizes the results by counting the number of rows for each combination of `lhs_cluster`, `lhs_database`, `rhs_cluster`, and `rhs_database`. ```kusto macro-expand MyEntityGroup_Outer as X From 49e4dd0495bf84d8fdb7017bd113d23abf33d3fd Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Mon, 27 Jan 2025 16:24:54 +0200 Subject: [PATCH 25/37] edits --- data-explorer/kusto/query/macro-expand-operator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index dd68170c90..dc4a556dfa 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -32,13 +32,13 @@ The `macro-expand` operator runs the subquery separately for each entity in the There are several syntactic variants that are used with the `macro-expand` operator: -1. **Explicit:** All elements are explicitly defined in the text of the operator invocation itself. +1. **Using the macro-expand operator:** All elements are explicitly defined in the text of the operator invocation itself. -1. **Explicit using let statement:** The entity group is specified in the query using a `let` statement outside the `macro-expand` operator using the syntax: +1. **Using a let statement:** The entity group is specified in the query using a `let` statement outside the `macro-expand` operator using the syntax: `let` *EntityGroupIdentifier* `=` `entity_group` `[`*EntityReference* [`,` ...]`]` -1. **Explicit using stored entity group:** The query uses an entity group stored in the database in scope rather than defined in the query. +1. **Using a stored entity group:** The query uses an entity group stored in the database in scope rather than defined in the query. To see examples of how the variants are used, see [Examples](#examples). From 8ace6cbf981d70d708525340e091fcbb72b0c759 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Mon, 27 Jan 2025 17:17:00 +0200 Subject: [PATCH 26/37] edits --- .../kusto/management/alter-entity-group.md | 12 ++++++------ .../kusto/management/alter-merge-entity-group.md | 12 ++++++------ .../kusto/management/create-entity-group.md | 14 +++++++------- .../kusto/management/drop-entity-group.md | 14 +++++++------- .../kusto/management/show-entity-group.md | 14 +++++++------- data-explorer/kusto/query/macro-expand-operator.md | 4 ++-- 6 files changed, 35 insertions(+), 35 deletions(-) diff --git a/data-explorer/kusto/management/alter-entity-group.md b/data-explorer/kusto/management/alter-entity-group.md index 7196c28549..1602cb5ad5 100644 --- a/data-explorer/kusto/management/alter-entity-group.md +++ b/data-explorer/kusto/management/alter-entity-group.md @@ -1,5 +1,5 @@ --- -title: .alter entity_group +title: .alter entity_group command description: Learn how to use the `.alter entity_group` command to change an existing entity group. ms.reviewer: ziham1531991 ms.topic: reference @@ -7,7 +7,7 @@ ms.date: 01/26/2025 --- -# .alter entity_group +# .alter entity_group command > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] @@ -58,7 +58,7 @@ The following example alters `MyEntityGroup` to include the entity, `cluster('c1 * [Entity groups](entity-groups.md) * [Entity types](../query/schema-entities/index.md) -* [.create entity_group](create-entity-group.md) -* [.alter-merge entity_group](alter-merge-entity-group.md) -* [.drop entity_group](drop-entity-group.md) -* [.show entity_group and .show entity_groups](show-entity-group.md) +* [.create entity_group command](create-entity-group.md) +* [.alter-merge entity_group command](alter-merge-entity-group.md) +* [.drop entity_group command](drop-entity-group.md) +* [.show entity_group(s) command](show-entity-group.md) diff --git a/data-explorer/kusto/management/alter-merge-entity-group.md b/data-explorer/kusto/management/alter-merge-entity-group.md index de4f0446af..86edd17d6e 100644 --- a/data-explorer/kusto/management/alter-merge-entity-group.md +++ b/data-explorer/kusto/management/alter-merge-entity-group.md @@ -1,12 +1,12 @@ --- -title: .alter-merge entity_group +title: .alter-merge entity_group command description: Learn how to use the `.alter-merge entity_group` command to change an existing entity group. ms.reviewer: ziham1531991 ms.topic: reference ms.date: 01/26/2025 --- -# .alter-merge entity_group +# .alter-merge entity_group command > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] @@ -70,7 +70,7 @@ Then run the following command to edit the existing entity group `MyEntityGroup` * [Entity groups](entity-groups.md) * [Entity types](../query/schema-entities/index.md) -* [.create entity_group](create-entity-group.md) -* [.alter entity_group](alter-entity-group.md) -* [.drop entity_group](drop-entity-group.md) -* [.show entity_group and .show entity_groups](show-entity-group.md) +* [.create entity_group command](create-entity-group.md) +* [.alter entity_group command](alter-entity-group.md) +* [.drop entity_group command](drop-entity-group.md) +* [.show entity_group(s) command](show-entity-group.md) diff --git a/data-explorer/kusto/management/create-entity-group.md b/data-explorer/kusto/management/create-entity-group.md index 9e3dc38b40..630c72530c 100644 --- a/data-explorer/kusto/management/create-entity-group.md +++ b/data-explorer/kusto/management/create-entity-group.md @@ -1,12 +1,12 @@ --- -title: .create entity_group(s) +title: .create entity_group command description: Learn how to use the `.create entity_group` command to create an entity group. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/26/2025 +ms.date: 01/27/2025 --- -# .create entity_group +# .create entity_group command > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] @@ -55,7 +55,7 @@ The following example creates the `MyEntityGroup` entity group with two entities * [Entity groups](entity-groups.md) * [Entity types](../query/schema-entities/index.md) -* [.alter entity_group](alter-entity-group.md) -* [.alter-merge entity_group](alter-merge-entity-group.md) -* [.drop entity_group](drop-entity-group.md) -* [.show entity_group and .show entity_groups](show-entity-group.md) \ No newline at end of file +* [.alter entity_group command](alter-entity-group.md) +* [.alter-merge entity_group command](alter-merge-entity-group.md) +* [.drop entity_group command](drop-entity-group.md) +* [.show entity_group(s) command](show-entity-group.md) \ No newline at end of file diff --git a/data-explorer/kusto/management/drop-entity-group.md b/data-explorer/kusto/management/drop-entity-group.md index f675710469..4d52e0f962 100644 --- a/data-explorer/kusto/management/drop-entity-group.md +++ b/data-explorer/kusto/management/drop-entity-group.md @@ -1,12 +1,12 @@ --- -title: .drop entity_group +title: .drop entity_group command description: Learn how to use the `.drop entity_group` command to remove an entity group from your database. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/26/2025 +ms.date: 01/27/2025 --- -# .drop entity_group +# .drop entity_group command > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] @@ -49,7 +49,7 @@ The following example drops the `MyEntityGroup` from your selected database. * [Entity groups](entity-groups.md) * [Entity types](../query/schema-entities/index.md) -* [.create entity_group](create-entity-group.md) -* [.alter entity_group](alter-entity-group.md) -* [.alter-merge entity_group](alter-merge-entity-group.md) -* [.show entity_group and .show entity_groups](show-entity-group.md) \ No newline at end of file +* [.create entity_group command](create-entity-group.md) +* [.alter entity_group command](alter-entity-group.md) +* [.alter-merge entity_group command](alter-merge-entity-group.md) +* [.show entity_group(s) command](show-entity-group.md) \ No newline at end of file diff --git a/data-explorer/kusto/management/show-entity-group.md b/data-explorer/kusto/management/show-entity-group.md index 06c68fa8a4..baffd49558 100644 --- a/data-explorer/kusto/management/show-entity-group.md +++ b/data-explorer/kusto/management/show-entity-group.md @@ -1,12 +1,12 @@ --- -title: .show entity_group(s) +title: .show entity_group(s) command description: Learn how to use the `.show entity_group` command to view existing entity groups. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/26/2025 +ms.date: 01/27/2025 --- -# .show entity_group(s) +# .show entity_group(s) command > [!INCLUDE [applies](../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../includes/applies-to-version/azure-data-explorer.md)] @@ -74,7 +74,7 @@ The following example returns the entity group, `eg1` along with its entity, `cl * [Entity groups](entity-groups.md) * [Entity types](../query/schema-entities/index.md) -* [.alter entity_group](alter-entity-group.md) -* [.alter-merge entity_group](alter-merge-entity-group.md) -* [.create entity_group](create-entity-group.md) -* [.drop entity_group](drop-entity-group.md) \ No newline at end of file +* [.alter entity_group command](alter-entity-group.md) +* [.alter-merge entity_group command](alter-merge-entity-group.md) +* [.create entity_group command](create-entity-group.md) +* [.drop entity_group command](drop-entity-group.md) \ No newline at end of file diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index dc4a556dfa..7191f1dc98 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -19,7 +19,7 @@ The `macro-expand` operator runs the subquery separately for each entity in the ## Syntax -`macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] *EntityGroup* `as` *EntityIdentifier* `(` *Subquery* `)` +`macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] *EntityGroup* `as` *EntityIdentifier* `(`*Subquery*`)` `macro-expand` [`kind` `=` *Kind*] [`isfuzzy` `=` *IsFuzzy*] `entity_group` `[`*EntityReference* [`,` ...] `as` *EntityIdentifier* `(`*Subquery*`)``]` @@ -159,6 +159,6 @@ macro-expand MyEntityGroup_Outer as X * [Entity groups](../management/entity-groups.md) * [join operator](join-operator.md) * [union operator](union-operator.md) -* [.show entity_group(s)](../management/show-entity-group.md) +* [.show entity_group(s) command](../management/show-entity-group.md) * [current_database()](current-database-function.md) * [current_cluster_endpoint()](current-cluster-endpoint-function.md) From 053973346f2efe6fc8727f205a3d5c70f0b8b193 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Tue, 28 Jan 2025 10:32:33 +0200 Subject: [PATCH 27/37] Update data-explorer/kusto/management/toc.yml Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/management/toc.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/data-explorer/kusto/management/toc.yml b/data-explorer/kusto/management/toc.yml index 9e27001792..20329efbce 100644 --- a/data-explorer/kusto/management/toc.yml +++ b/data-explorer/kusto/management/toc.yml @@ -260,19 +260,19 @@ items: - name: Entity Group href: entity-groups.md - name: .create entity_group command - displayName: .create entity_group + displayName: create entity_group href: create-entity-group.md - name: .alter entity_group command - displayName: .alter entity_group + displayName: alter entity_group href: alter-entity-group.md - name: .alter-merge entity_group command - displayName: .alter-merge entity_group + displayName: alter-merge entity_group href: alter-merge-entity-group.md - name: .drop entity_group command - displayName: .drop entity_group + displayName: drop entity_group href: drop-entity-group.md - name: .show entity_group command - displayName: .show entity_group + displayName: show entity_group, show entity_groups, .show entity_group href: show-entity-group.md - name: Functions items: From 05ee044054a87d04ad699b818ca5f477a48e1a14 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Tue, 28 Jan 2025 10:33:54 +0200 Subject: [PATCH 28/37] Update data-explorer/kusto/query/macro-expand-operator.md Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/query/macro-expand-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 7191f1dc98..e8010554cd 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -13,7 +13,7 @@ The `macro-expand` operator simplifies running a subquery on a set of [entities] The operator is useful when you have data spread across multiple clusters, databases, or tables. For example, when data is held in the same global region as its source, you can use the `macro-expand` operator in a single query across data in distinct locations, instead of running separate queries for each entity and combining the results manually. -The set of entities you want to query is called an **entity group**. The entity group can be stored for reuse in a database or defined directly in your query text. For more information about stored entity groups, see [Entity groups](../management/entity-groups.md). +The set of entities you want to query is called an *entity group*. The entity group can be stored for reuse in a database or defined directly in your query text. For more information about stored entity groups, see [Entity groups](../management/entity-groups.md). The `macro-expand` operator runs the subquery separately for each entity in the group and then combines all the results into a single output. The subquery can include nested `macro-expand` operators. However, the identifier for the inner `macro-expand` must be different from the identifier for the outer `macro-expand` to clearly distinguish between the scope and references of each one. From 8ed7eb6ae17ea709174408e945f16a76293945c8 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Wed, 29 Jan 2025 14:21:36 +0200 Subject: [PATCH 29/37] edits --- .../kusto/management/show-entity-group.md | 2 ++ .../kusto/query/macro-expand-operator.md | 34 ++++++++++--------- 2 files changed, 20 insertions(+), 16 deletions(-) diff --git a/data-explorer/kusto/management/show-entity-group.md b/data-explorer/kusto/management/show-entity-group.md index baffd49558..873659a515 100644 --- a/data-explorer/kusto/management/show-entity-group.md +++ b/data-explorer/kusto/management/show-entity-group.md @@ -41,6 +41,8 @@ This command returns a table with the following columns: ## Examples +The following examples show how to use the `.show entity_group` and `.show entity_group commands`. + ### Show entity groups The following example returns all the entity groups in the selected database, `eg1` and `eg2`, along with their entities. diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index e8010554cd..7c08929b0a 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -28,19 +28,21 @@ The `macro-expand` operator runs the subquery separately for each entity in the > [!NOTE] > The operation of the `macro-expand` operator can be modified by setting the `best_effort` request property to `true`, either by using a `set statement` or through [client request properties](../api/rest/request-properties.md). When this property is set to `true`, the `macro-expand` operator ignores fuzzy resolution and connectivity failures, to execute any of the subexpressions being unioned and issues a warning in the query status results. -### Syntax variations +### Variations -There are several syntactic variants that are used with the `macro-expand` operator: +There are several ways to use the `macro-expand` operator: -1. **Using the macro-expand operator:** All elements are explicitly defined in the text of the operator invocation itself. +1. **Inline:** All elements are explicitly defined in the text of the operator invocation itself. For an example, see [Calculate errors](#calculate-errors). -1. **Using a let statement:** The entity group is specified in the query using a `let` statement outside the `macro-expand` operator using the syntax: +1. **Via `let` statement:** The entity group is specified in the query using a `let` statement outside the `macro-expand` operator using the syntax: `let` *EntityGroupIdentifier* `=` `entity_group` `[`*EntityReference* [`,` ...]`]` + For an example, see [Calculate SKU errors using `let` statement](#calculate-sku-errors-using-let-statement). + 1. **Using a stored entity group:** The query uses an entity group stored in the database in scope rather than defined in the query. -To see examples of how the variants are used, see [Examples](#examples). + For an example, see [Extend table with contextual scalar functions](#extend-table-with-contextual-scalar-functions). ## Parameters @@ -67,12 +69,12 @@ The `macro-expand` subquery can reference two specialized scalar functions as if ### Calculate errors -The following example uses an entity group to calculate the number of errors produced by each Stock Keeping Unit (SKU). It defines an `entity_group`, `X`, that includes databases named `Kuskus` in two clusters. The query then performs a subquery to filter for error logs and counts the errors by `Source`. Next it performs an `inner` join on `Source` with the `DimCluster` table to get the `SKU` for each source. Finally, it sums the error counts by `SKU`. +The following example uses an entity group, per the [inline variation](#variations), to calculate the number of errors produced by each Stock Keeping Unit (SKU). It defines an `entity_group`, `X`, that includes databases named `MyDatabase` in two clusters. The query then performs a subquery to filter for error logs and counts the errors by `Source`. Next it performs an `inner` join on `Source` with the `DimCluster` table to get the `SKU` for each source. Finally, it sums the error counts by `SKU`. ```kusto -macro-expand entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')] as X +macro-expand entity_group [cluster('C1').database('MyDatabase'), cluster('C2').database('MyDatabase')] as X ( - X.KustoLogs + X.Logs | where Level == 'Error' | summarize Count=count() by Source | join kind=inner (X.DimCluster | project SKU, Source) on Source @@ -89,27 +91,27 @@ To write the same query without using `macro-expand`, it might look as follows: ```kusto union ( - cluster('C1').database('Kuskus').KustoLogs + cluster('C1').database('MyDatabase').KustoLogs | where Level == 'Error' | summarize Count=count() by Source - | join kind=inner (cluster('C1').database('Kuskus').DimCluster | project SKU, Source) on Source + | join kind=inner (cluster('C1').database('MyDatabase').DimCluster | project SKU, Source) on Source ), ( - cluster('C2').database('Kuskus').KustoLogs + cluster('C2').database('MyDatabase').KustoLogs | where Level == 'Error' | summarize Count=count() by Source - | join kind=inner (cluster('C2').database('Kuskus').DimCluster | project SKU, Source) on Source + | join kind=inner (cluster('C2').database('MyDatabase').DimCluster | project SKU, Source) on Source ) | summarize Sum=sum(Count) by SKU ``` ### Calculate SKU errors using `let` statement -The following example uses a `let` statement to define an entity group named `UberKuskus` which includes the `Kuskus` database from both `C1` and `C2` clusters. This entity group is then used to perform the same query in the [previous example](#calculate-errors) to calculate the number of errors produced by each SKU. The `macro-expand` operator is used to reference the `UberKuskus` entity group (alias `X`). +The following example uses a [`let` statement](#variations) to define an entity group named `Greater` which includes the `MyDatabase` database from both `C1` and `C2` clusters. This entity group is then used to perform the same query in the [previous example](#calculate-errors) to calculate the number of errors produced by each SKU. The `macro-expand` operator is used to reference the `Greater` entity group (alias `X`). ```kusto -let UberKuskus = entity_group [cluster('C1').database('Kuskus'), cluster('C2').database('Kuskus')]; -macro-expand UberKuskus as X +let GreaterDatabase = entity_group [cluster('C1').database('MyDatabase'), cluster('C2').database('MyDatabase')]; +macro-expand GreaterDatabase as X ( X.KustoLogs | where Level == 'Error' @@ -121,7 +123,7 @@ macro-expand UberKuskus as X ### Extend table with contextual scalar functions -The following query runs a subquery on the `Admins` table from each entity using [a stored entity group](../management/create-entity-group.md#examples), `MyEntityGroup`. It uses `$current_database` and `$current_cluster_endpoint` to extend the table, adding the current database and current cluster for each row. Then, it summarizes the results by counting the number of rows for each combination of `current_cluster` and `current_database`. +The following query runs a subquery on the `Admins` table from each entity using the [stored entity group](#variations) variation, with the `MyEntityGroup`. For more information on how to create a stored entity, see [.create entity_group command](../management/create-entity-group.md). It uses `$current_database` and `$current_cluster_endpoint` to extend the table, adding the current database and current cluster for each row. Then, it summarizes the results by counting the number of rows for each combination of `current_cluster` and `current_database`. ```kusto macro-expand MyEntityGroup as X From 81756d487a56e5d32e339f8420b08c9128148dcf Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Wed, 29 Jan 2025 14:36:39 +0200 Subject: [PATCH 30/37] edits --- data-explorer/kusto/query/macro-expand-operator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 7c08929b0a..2fa38bdb68 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -91,13 +91,13 @@ To write the same query without using `macro-expand`, it might look as follows: ```kusto union ( - cluster('C1').database('MyDatabase').KustoLogs + cluster('C1').database('MyDatabase').Logs | where Level == 'Error' | summarize Count=count() by Source | join kind=inner (cluster('C1').database('MyDatabase').DimCluster | project SKU, Source) on Source ), ( - cluster('C2').database('MyDatabase').KustoLogs + cluster('C2').database('MyDatabase').Logs | where Level == 'Error' | summarize Count=count() by Source | join kind=inner (cluster('C2').database('MyDatabase').DimCluster | project SKU, Source) on Source @@ -113,7 +113,7 @@ The following example uses a [`let` statement](#variations) to define an entity let GreaterDatabase = entity_group [cluster('C1').database('MyDatabase'), cluster('C2').database('MyDatabase')]; macro-expand GreaterDatabase as X ( - X.KustoLogs + X.Logs | where Level == 'Error' | summarize Count=count() by Source | join kind=inner (X.DimCluster | project SKU, Source) on Source From dfc441f8486f7a845267adf1b1de217d48e1be5e Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Thu, 30 Jan 2025 14:12:33 +0200 Subject: [PATCH 31/37] Update data-explorer/kusto/query/macro-expand-operator.md Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/query/macro-expand-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 2fa38bdb68..8d902c3ca6 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -69,7 +69,7 @@ The `macro-expand` subquery can reference two specialized scalar functions as if ### Calculate errors -The following example uses an entity group, per the [inline variation](#variations), to calculate the number of errors produced by each Stock Keeping Unit (SKU). It defines an `entity_group`, `X`, that includes databases named `MyDatabase` in two clusters. The query then performs a subquery to filter for error logs and counts the errors by `Source`. Next it performs an `inner` join on `Source` with the `DimCluster` table to get the `SKU` for each source. Finally, it sums the error counts by `SKU`. +The following example uses an [inline variation](#variations) entity group to calculate the number of errors produced by each Stock Keeping Unit (SKU). It defines an `entity_group`, `X`, that includes databases named `MyDatabase` in two clusters. The query then performs a subquery to filter for error logs and counts the errors by `Source`. Next it performs an `inner` join on `Source` with the `DimCluster` table to get the `SKU` for each source. Finally, it sums the error counts by `SKU`. ```kusto macro-expand entity_group [cluster('C1').database('MyDatabase'), cluster('C2').database('MyDatabase')] as X From 7b55419f0ba4cbf0445d4fae3f01384f80663dac Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Thu, 30 Jan 2025 14:13:33 +0200 Subject: [PATCH 32/37] Update data-explorer/kusto/query/macro-expand-operator.md Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/query/macro-expand-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 8d902c3ca6..6550a49f3f 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -107,7 +107,7 @@ union ### Calculate SKU errors using `let` statement -The following example uses a [`let` statement](#variations) to define an entity group named `Greater` which includes the `MyDatabase` database from both `C1` and `C2` clusters. This entity group is then used to perform the same query in the [previous example](#calculate-errors) to calculate the number of errors produced by each SKU. The `macro-expand` operator is used to reference the `Greater` entity group (alias `X`). +The following example uses a [`let` statement](#variations) to define an entity group in a variable named `Greater` which includes the `MyDatabase` database from both `C1` and `C2` clusters. This entity group is then used to perform the same query in the [previous example](#calculate-errors) to calculate the number of errors produced by each SKU. The `macro-expand` operator is used to reference the `Greater` entity group (alias `X`). ```kusto let GreaterDatabase = entity_group [cluster('C1').database('MyDatabase'), cluster('C2').database('MyDatabase')]; From 59c490cf79b34c31deb3111fff1989adfd644184 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Thu, 30 Jan 2025 14:13:59 +0200 Subject: [PATCH 33/37] Update data-explorer/kusto/query/macro-expand-operator.md Co-authored-by: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> --- data-explorer/kusto/query/macro-expand-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 6550a49f3f..1e659f9907 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -123,7 +123,7 @@ macro-expand GreaterDatabase as X ### Extend table with contextual scalar functions -The following query runs a subquery on the `Admins` table from each entity using the [stored entity group](#variations) variation, with the `MyEntityGroup`. For more information on how to create a stored entity, see [.create entity_group command](../management/create-entity-group.md). It uses `$current_database` and `$current_cluster_endpoint` to extend the table, adding the current database and current cluster for each row. Then, it summarizes the results by counting the number of rows for each combination of `current_cluster` and `current_database`. +The following query runs a subquery on the `Admins` table from each entity using the [stored entity group](#variations) variation, using the stored entity group `MyEntityGroup`. For more information on how to create a stored entity, see [.create entity_group command](../management/create-entity-group.md). It uses `$current_database` and `$current_cluster_endpoint` to extend the table, adding the current database and current cluster for each row. Then, it summarizes the results by counting the number of rows for each combination of `current_cluster` and `current_database`. ```kusto macro-expand MyEntityGroup as X From d7b14a10748c8a1640af6ce06b52e891e0c82b41 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Thu, 30 Jan 2025 14:16:24 +0200 Subject: [PATCH 34/37] Update data-explorer/kusto/query/macro-expand-operator.md --- data-explorer/kusto/query/macro-expand-operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index 1e659f9907..fea6c3f1ef 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -30,7 +30,7 @@ The `macro-expand` operator runs the subquery separately for each entity in the ### Variations -There are several ways to use the `macro-expand` operator: + There are several ways to specify the entity group used by the `macro-expand` operator: 1. **Inline:** All elements are explicitly defined in the text of the operator invocation itself. For an example, see [Calculate errors](#calculate-errors). From df5a5b07d661c92dcca35df68a85fb58fc2e3bc5 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Thu, 30 Jan 2025 16:10:28 +0200 Subject: [PATCH 35/37] edits --- data-explorer/kusto/query/macro-expand-operator.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index fea6c3f1ef..ea2a210c2f 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -3,7 +3,7 @@ title: macro-expand operator description: Learn how to use the macro-expand operator to run a subquery on a set of entities. ms.reviewer: ziham1531991 ms.topic: reference -ms.date: 01/27/2025 +ms.date: 01/30/2025 --- # macro-expand operator @@ -30,7 +30,7 @@ The `macro-expand` operator runs the subquery separately for each entity in the ### Variations - There are several ways to specify the entity group used by the `macro-expand` operator: +There are several ways to specify the entity group used by the `macro-expand` operator: 1. **Inline:** All elements are explicitly defined in the text of the operator invocation itself. For an example, see [Calculate errors](#calculate-errors). @@ -67,6 +67,8 @@ The `macro-expand` subquery can reference two specialized scalar functions as if ## Examples +The following examples show how to use the `macro-expand`operator. + ### Calculate errors The following example uses an [inline variation](#variations) entity group to calculate the number of errors produced by each Stock Keeping Unit (SKU). It defines an `entity_group`, `X`, that includes databases named `MyDatabase` in two clusters. The query then performs a subquery to filter for error logs and counts the errors by `Source`. Next it performs an `inner` join on `Source` with the `DimCluster` table to get the `SKU` for each source. Finally, it sums the error counts by `SKU`. @@ -123,7 +125,7 @@ macro-expand GreaterDatabase as X ### Extend table with contextual scalar functions -The following query runs a subquery on the `Admins` table from each entity using the [stored entity group](#variations) variation, using the stored entity group `MyEntityGroup`. For more information on how to create a stored entity, see [.create entity_group command](../management/create-entity-group.md). It uses `$current_database` and `$current_cluster_endpoint` to extend the table, adding the current database and current cluster for each row. Then, it summarizes the results by counting the number of rows for each combination of `current_cluster` and `current_database`. +The following query uses the [stored entity group](#variations) variation. It runs a subquery on the `Admins` table from each entity using the stored entity group `MyEntityGroup`. For more information on how to create a stored entity, see [.create entity_group command](../management/create-entity-group.md). It uses `$current_database` and `$current_cluster_endpoint` to extend the table, adding the current database and current cluster for each row. Then, it summarizes the results by counting the number of rows for each combination of `current_cluster` and `current_database`. ```kusto macro-expand MyEntityGroup as X From ec1e22961e9923ac810c41c630533947e167d9e7 Mon Sep 17 00:00:00 2001 From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com> Date: Sun, 2 Feb 2025 12:28:39 +0200 Subject: [PATCH 36/37] Apply suggestions from code review --- data-explorer/kusto/management/show-entity-group.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/kusto/management/show-entity-group.md b/data-explorer/kusto/management/show-entity-group.md index 873659a515..edf67c2c8d 100644 --- a/data-explorer/kusto/management/show-entity-group.md +++ b/data-explorer/kusto/management/show-entity-group.md @@ -41,7 +41,7 @@ This command returns a table with the following columns: ## Examples -The following examples show how to use the `.show entity_group` and `.show entity_group commands`. +The following examples show how to use the `.show entity_group` and `.show entity_group` commands. ### Show entity groups From bfc2639c7ecfc0cf91b3fe28b6d3d8689c17ee69 Mon Sep 17 00:00:00 2001 From: Tracey Torble <56347952+ttorble@users.noreply.github.com> Date: Mon, 3 Feb 2025 09:32:28 +0000 Subject: [PATCH 37/37] Update data-explorer/kusto/query/macro-expand-operator.md --- data-explorer/kusto/query/macro-expand-operator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/data-explorer/kusto/query/macro-expand-operator.md b/data-explorer/kusto/query/macro-expand-operator.md index ea2a210c2f..7e8417277d 100644 --- a/data-explorer/kusto/query/macro-expand-operator.md +++ b/data-explorer/kusto/query/macro-expand-operator.md @@ -32,15 +32,15 @@ The `macro-expand` operator runs the subquery separately for each entity in the There are several ways to specify the entity group used by the `macro-expand` operator: -1. **Inline:** All elements are explicitly defined in the text of the operator invocation itself. For an example, see [Calculate errors](#calculate-errors). +- **Inline:** All elements are explicitly defined in the text of the operator invocation itself. For an example, see [Calculate errors](#calculate-errors). -1. **Via `let` statement:** The entity group is specified in the query using a `let` statement outside the `macro-expand` operator using the syntax: +- **Via `let` statement:** The entity group is specified in the query using a `let` statement outside the `macro-expand` operator using the syntax: `let` *EntityGroupIdentifier* `=` `entity_group` `[`*EntityReference* [`,` ...]`]` For an example, see [Calculate SKU errors using `let` statement](#calculate-sku-errors-using-let-statement). -1. **Using a stored entity group:** The query uses an entity group stored in the database in scope rather than defined in the query. +- **Using a stored entity group:** The query uses an entity group stored in the database in scope rather than defined in the query. For an example, see [Extend table with contextual scalar functions](#extend-table-with-contextual-scalar-functions).