Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
6912d15
Add Linux Container install instructions
vplauzon May 9, 2023
1d11d30
.
vplauzon Jan 17, 2025
013dbfd
First draft for Linux-focus how-to
vplauzon Jan 17, 2025
a01d69e
Merge
vplauzon Jan 17, 2025
2957b86
Fix merge issues
vplauzon Jan 17, 2025
d67a84e
replace branch PR 6555 (4)
ktalmor Jan 26, 2025
d84420f
Merge branch 'main' of https://github.com/ktalmor/dataexplorer-docs-p…
ktalmor Jan 28, 2025
85cff7e
Added example descriptions
ktalmor Jan 28, 2025
96c213b
deleted sentence
ktalmor Jan 29, 2025
2732d52
Merge branch 'main' of https://github.com/MicrosoftDocs/dataexplorer-…
ktalmor Jan 29, 2025
573c60d
Merge branch 'main' of https://github.com/MicrosoftDocs/dataexplorer-…
ktalmor Feb 5, 2025
3e3381e
Added Examples introductory sentence
ktalmor Feb 5, 2025
370bd18
Merge branch 'vpl/emulator-linux' of https://github.com/vplauzon/data…
mjosephym Feb 5, 2025
c52bf54
(AzureCXP) fixes MicrosoftDocs//dataexplorer-docs#367732
PesalaPavan Feb 6, 2025
c1ae500
Remove `monikerRange` from visualization articles
ktalmor Feb 6, 2025
7507c4e
Emulator updates
mjosephym Feb 6, 2025
46a587c
Merge branch 'KQL-review-batch-11' of https://github.com/ktalmor/data…
ktalmor Feb 6, 2025
5ba4f0f
tabs
mjosephym Feb 6, 2025
b806eef
edits
mjosephym Feb 6, 2025
17996a3
edits
mjosephym Feb 6, 2025
c7dbf14
edits
mjosephym Feb 6, 2025
daf4a15
edits
mjosephym Feb 6, 2025
e8c3e40
edits
mjosephym Feb 6, 2025
04781cd
edits
mjosephym Feb 6, 2025
a09d9d5
edit
mjosephym Feb 9, 2025
3f0e365
edit
mjosephym Feb 9, 2025
d1dd64b
New include for ADX only help cluster
ktalmor Feb 11, 2025
ef2e297
Include for ADX only help cluster
ktalmor Feb 11, 2025
7d04312
Update alter-merge-managed-identity-policy-command.md
slavikn84 Feb 11, 2025
3c61750
Update managed identity policy command documentation.
slavikn84 Feb 11, 2025
dc0f8dd
Update data-explorer/kusto/management/alter-merge-managed-identity-po…
slavikn84 Feb 12, 2025
001cc92
Update data-explorer/kusto/management/alter-merge-managed-identity-po…
slavikn84 Feb 12, 2025
ee72f02
Update data-explorer/kusto/management/alter-merge-managed-identity-po…
slavikn84 Feb 12, 2025
b3524cf
Update data-explorer/kusto/management/alter-merge-managed-identity-po…
slavikn84 Feb 12, 2025
537c77a
Apply suggestions from code review
shsagir Feb 12, 2025
9c70338
Update data-explorer/kusto/management/alter-merge-managed-identity-po…
shsagir Feb 12, 2025
736eb95
Update data-explorer/kusto/management/alter-merge-managed-identity-po…
shsagir Feb 12, 2025
7212a1d
Merge pull request #6631 from slavikn84/patch-32
prmerger-automator[bot] Feb 12, 2025
c09549b
Update project-rename-operator.md example and output
shsagir Feb 12, 2025
b920b78
Merge branch 'main' into patch-2
shsagir Feb 12, 2025
c826c3f
Apply suggestions from code review
shsagir Feb 12, 2025
7ff398e
Fix broken links in Kusto emulator install doc
shsagir Feb 12, 2025
48d5ad5
Apply suggestions from code review
shsagir Feb 12, 2025
1de31a8
Merge pull request #6614 from PesalaPavan/patch-2
prmerger-automator[bot] Feb 12, 2025
572279c
renamed file
ktalmor Feb 12, 2025
04cafa7
replaced link to renamed include file
ktalmor Feb 12, 2025
0fafddc
Merge pull request #6617 from mjosephym/370392-emulator
Stacyrch140 Feb 12, 2025
67640db
Merge pull request #6584 from ktalmor/KQL-review-batch-11
Stacyrch140 Feb 12, 2025
bcdcfab
Merge pull request #6633 from MicrosoftDocs/main
Taojunshen Feb 12, 2025
bb82719
Merging changes synced from https://github.com/MicrosoftDocs/dataexpl…
Feb 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions data-explorer/includes/kusto-emulator-limitations.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
ms.service: azure
ms.topic: include
ms.date: 08/14/2022
ms.date: 02/05/2025
---

## Limitations
Expand All @@ -21,7 +21,7 @@ In general, this offering isn't suited for production workloads.
- No support for ingestion endpoints, including Kusto.Ingest SDKs
- No streaming ingestion
- Although ingested data can be stored externally to the container, we don't recommend persisting data for long periods of time for the following reasons:
- There's no guarantee the [extent](/kusto/management/extents-overview?view=azure-data-explorer&preserve-view=true) format will be compatible between versions of the Kusto emulator
- There's no guarantee the [extent](/kusto/management/extents-overview?view=azure-data-explorer&preserve-view=true) format is compatible between versions of the Kusto emulator
- Extents aren't [merged](/kusto/management/merge-policy?view=azure-data-explorer&preserve-view=true) and therefore can become fragmented as data get ingested
- Retention policies can be set but won't be honored
- Retention and partitioning policies can be set but aren't honored
- The [Python plugin](/kusto/query/python-plugin?view=azure-data-explorer&preserve-view=true?view=azure-data-explorer&preserve-view=true) isn't supported
88 changes: 55 additions & 33 deletions data-explorer/kusto-emulator-install.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,30 @@
---
title: Install the Azure Data Explorer Kusto emulator
description: In this article, you'll learn how to install the Azure Data Explorer Kusto emulator and run your first query.
description: In this article, you learn how to install the Azure Data Explorer Kusto emulator and run your first query.
ms.reviewer: vplauzon
ms.topic: how-to
ms.date: 12/05/2023
ms.date: 02/09/2025
---

# Install the Azure Data Explorer Kusto emulator

You can install the Azure Data Explorer Kusto emulator in the following ways:

- On your own device: Consider using this option if you need to provision a local development environment
- On a CI/CD agent virtual machine (VM): Use this option if you require a CI/CD pipeline for running automated tests
- **On your own device**: Consider using this option if you need to provision a local development environment
- **On a CI/CD agent virtual machine (VM)**: Use this option if you require a CI/CD pipeline for running automated tests

The emulator is available as a *Windows* or *Linux* Docker container image.
The emulator is available as a *Linux* and *Windows* Docker container image.

In this article, you'll learn how to:
In this article, you learn how to:

- [Install the Kusto emulator](#install-the-kusto-emulator)
- [Connect to the emulator](#connect-to-the-emulator)
- [Create a database](#create-a-database)
- [Ingest data](#ingest-data)
- [Query data](#query-data)

This article focuses on how to install the Linux Docker container on a Windows client.

## Prerequisites

- The host operating system must be one of:
Expand All @@ -35,25 +37,35 @@ In this article, you'll learn how to:
> Linux distros only support Linux container images.

- 2 gigabytes (GB) of RAM minimum; we recommend using 4 GB or more
- Docker Client for [Windows](https://docs.docker.com/desktop/windows/install/) or [Linux](https://docs.docker.com/desktop/install/linux-install/)
- Docker Client for [Linux](https://docs.docker.com/desktop/install/linux-install/) or [Windows](https://docs.docker.com/desktop/windows/install/)

## Install the Kusto emulator

The following steps are for using PowerShell to start the emulator using the [Kusto emulator container image](https://aka.ms/adx.emulator.image). For other options, see [Run emulator options](#run-emulator-options).

1. For Windows only, switch Docker to run with Windows containers. You may need to enable the feature in the Docker settings.
1. **For Windows container only**: Switch Docker to run with Windows containers. You might need to enable the feature in the Docker settings.

:::image type="content" source="media/kusto-emulator/kusto-emulator-docker-windows-container.png" alt-text="Screenshot of the Docker settings, showing the Switch to Windows containers option.":::

1. Run the following command to start the emulator.

> [!IMPORTANT]
> The Kusto emulator container image is a free offering under the [Microsoft Software License Terms](https://aka.ms/adx.emulator.license). Since the emulator runs in a container, you must accept the license terms by passing the `ACCEPT_EULA` environment variable to the container with its value set to `Y` indicating.
> The Kusto emulator container image is a free offering under the [Microsoft Software License Terms](https://aka.ms/adx.emulator.license). Since the emulator runs in a container, you must accept the license terms by passing the `ACCEPT_EULA` environment variable to the container with its value set to `Y`.

> [!NOTE]
>
> - The first time this command is run, Docker pulls the container image which is several GBs in size and may take several minutes to download. Once downloaded, the image is cached and available for subsequent runs without having to download it again.
> - (For Windows container only) The container must be run in process-isolation mode. This is the default on some versions of Docker. For other versions, you can start the container in Hyper-V isolation mode by adding `--isolation=hyperv` to the run command.
> - The first time this command is run, Docker pulls the container image which is several GBs in size and might take several minutes to download. Once downloaded, the image is cached and available for subsequent runs without having to download it again.
> - **For Windows container only**: The container must run in process-isolation mode. This is the default on some versions of Docker. For other versions, you can start the container in Hyper-V isolation mode by adding `--isolation=hyperv` to the run command.

# [Linux container](#tab/linux-container)

To start the Linux container, make sure you use the `latest` or `stable` tag:

```powershell
docker run -e ACCEPT_EULA=Y -m 4G -d -p 8080:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest
```

# [Windows container](#tab/windowscontainer)

- To start the Windows container on Windows Server operating system, make sure you use the `latest` or `stable` tag:

Expand All @@ -67,11 +79,7 @@ The following steps are for using PowerShell to start the emulator using the [Ku
docker run -e ACCEPT_EULA=Y -m 4G -d -p 8080:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer:windows11
```

- To start the Linux container:

```powershell
docker run -e ACCEPT_EULA=Y -m 4G -d -p 8080:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest
```
---

1. Run the following command to verify that the container is running.

Expand Down Expand Up @@ -114,24 +122,35 @@ The following steps are for using PowerShell to start the emulator using the [Ku
RawContentLength : 988
```

### Run emulator options
## Run emulator options

You can use any of the following options when running the emulator:

- Mount a local folder to the container: Use this option to mount a folder in the host environment into the container. Mounting a host folder enables your queries to interact with local files, which is useful for [creating a database persistent between container runs](#create-a-database) and [ingesting data](#ingest-data).

For example, to mount the folder "D:\host\local" on the host to the folder "c:\kustodatadata" in the container, use the next command on Windows Server:
# [Linux container](#tab/linux-container)

For example, to mount the folder "D:\host\local" on the host to the folder "/kustodatadata" in the container, use the following command on Windows Server:

```powershell
docker run -v d:\host\local:/kustodata -e ACCEPT_EULA=Y -m 4G -d -p 8080:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest
```

# [Windows container](#tab/windowscontainer)
For example, to mount the folder "D:\host\local" on the host to the folder "c:\kustodata" in the container, use the following command on Windows Server:

```powershell
docker run -v d:\host\local:c:\kustodata -e ACCEPT_EULA=Y -m 4G -d -p 8080:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer:latest
```

---

- Run on a different port: The Kusto emulator exposes access to the Kusto Query Engine on port 8080; hence in other examples you mapped the host port 8080 to the emulator port 8080. You can use this option to map a different host to the engine.

For example, to map port 9000 on the host to the engine, use the following command on Windows Server:

```powershell
docker run -e ACCEPT_EULA=Y -m 4G -d -p 9000:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer:latest
docker run -e ACCEPT_EULA=Y -m 4G -d -p 9000:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest
```

## Connect to the emulator
Expand All @@ -142,55 +161,58 @@ You can use any of the following tools to connect to and interact with the emula
- [Kusto.CLI](/kusto/tools/kusto-cli?view=azure-data-explorer&preserve-view=true)
- [Kusto.Data SDKs](/kusto/api/netfx/about-kusto-data?view=azure-data-explorer&preserve-view=true)

In the following sections, you'll use Kusto.Explorer to create a database, ingest data, and query it. To learn more, see [Using Kusto.Explorer](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true).
In the following sections, you use Kusto.Explorer to create a database, ingest data, and query it. To learn more, see [Using Kusto.Explorer](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true).

> [!IMPORTANT]
> The Kusto Emulator doesn't support HTTPS or Microsoft Entra authentication.
>
> You must first allow unsafe connections in **Tools** > **Options** > **Connections** > **Allow unsafe connections**.
>
>
> :::image type="content" source="media/kusto-emulator/unsafe-connections.png" alt-text="Screenshot of Kusto Explorer connection options.":::
>
>
> The following image highlights the affected fields in the **Add connection** properties.
>
> - The **Cluster connection** must begin with `http://` and not `https://`.
> - In **Security** > **Advanced: Connection String**, you'll need to remove the `AAD Federated Security=True` portion of the connection string to disable Microsoft Entra authentication.
> - In **Security** > **Advanced: Connection String**, you need to remove the `AAD Federated Security=True` portion of the connection string to disable Microsoft Entra authentication.
>
> :::image type="content" source="media/kusto-emulator/kusto-emulator-connection.png" alt-text="Screenshot of Kusto Explorer single connection setup.":::

## Create a database

To store data and run queries, create a database or attach the emulator to an existing database.

A database can be persisted in a container folder or on a [mounted folder](#run-emulator-options). The former's lifetime is bound to the container, so restarting the container loses any changes. Also, the container virtual storage is less efficient than native one. Mounted folder enables you to keep the data between container runs.
A database can be persisted in a container folder or on a [mounted folder](#run-emulator-options). The former's lifetime is bound to the container, so restarting the container loses any changes. Also, the container virtual storage is less efficient than native one. Mounted folder enables you to keep the data between container runs.

In this example, we keep the data on the container.

In the [Kusto.Explorer Query mode](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true#query-mode), run the following command to create a persistent database if using a Windows container image:
In the [Kusto.Explorer Query mode](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true#query-mode), run the following command to create a persistent database:

# [Linux container](#tab/linux-container)

```kusto
.create database <YourDatabaseName> persist (
@"c:\kustodata\dbs\<YourDatabaseName>\md",
@"c:\kustodata\dbs\<YourDatabaseName>\data"
@"/kustodata/dbs/<YourDatabaseName>/md",
@"/kustodata/dbs/<YourDatabaseName>/data"
)
```

The equivalent command for a Linux container image is:
# [Windows container](#tab/windowscontainer)

```kusto
.create database <YourDatabaseName> persist (
@"/kustodata/dbs/<YourDatabaseName>/md",
@"/kustodata/dbs/<YourDatabaseName>/data"
@"c:\kustodata\dbs\<YourDatabaseName>\md",
@"c:\kustodata\dbs\<YourDatabaseName>\data"
)
```
---

This command requires that the folders don't already exist, to prevent over-writing existing information. To attach to an existing database, use the following command instead, specifying the path that ends with `md`:

```kusto
.attach database <YourDatabaseName> from @"<PathToMdFolder>"
```

It's also possible to detach the database from the emulator, which will keep all the database metadata and data intact (so you could reattach to it in the future):
It's also possible to detach the database from the emulator, which will keep all the database metadata and data intact so you can reattach to it in the future:

```kusto
.detach database <YourDatabaseName>
Expand All @@ -200,7 +222,7 @@ It's also possible to detach the database from the emulator, which will keep all

To ingest data, you must first create an external table linked to a file and then ingest the data into a table in the database.

Use the steps in the following example to create an external table and ingest data into it. For the example, in the local folder *c:\kustodata*, create a file called `sample.csv` with the following data:
Use the steps in the following example to create an external table and ingest data into it. As part of the example, create a file called `sample.csv` in the local folder *d:\host\local*, with the following data:

```text
Alice, 1
Expand All @@ -217,7 +239,7 @@ Carl, 3
1. Run the following command to [ingest the file into the table](/kusto/management/data-ingestion/ingest-from-storage?view=azure-data-explorer&preserve-view=true):

```kusto
.ingest into table MyIngestedSample(@"c:\kustodata\sample.csv")
.ingest into table MyIngestedSample(@"/kustodata/sample.csv")
```

## Query data
Expand Down
14 changes: 7 additions & 7 deletions data-explorer/kusto-emulator-overview.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
title: Azure Data Explorer Kusto emulator overview
description: In this article, you'll learn about the Azure Data Explorer Kusto emulator
description: In this article, you learn about the Azure Data Explorer Kusto emulator.
ms.reviewer: vplauzon
ms.topic: conceptual
ms.date: 05/08/2023
ms.date: 02/06/2025
---
# Azure Data Explorer Kusto emulator

Expand All @@ -13,11 +13,11 @@ The Kusto emulator is a local environment that encapsulates the query engine. Yo
>
> - The emulator is provided *as-is*, without any support or warranties.
> - The emulator isn't intended for use in production environments.
> - The [license terms](https://aka.ms/adx.emulator.license) explicitly prohibit benchmark tests using the emulator as it is not intended or optimized for that purpose. The emulator has a very different performance profile compared to the Azure Data Explorer service.
> - The [license terms](https://aka.ms/adx.emulator.license) explicitly prohibit benchmark tests using the emulator as it isn't intended or optimized for that purpose. The emulator has a different performance profile compared to the Azure Data Explorer service.

## Architecture

The emulator is available as a *Windows* or *Linux* Docker container image. It exposes a query endpoint over HTTP that can be used with any client including [Kusto.Explorer](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true), [Kusto.CLI](/kusto/tools/kusto-cli?view=azure-data-explorer&preserve-view=true), or [Kusto.Data SDKs](/kusto/api/netfx/about-kusto-data?view=azure-data-explorer&preserve-view=true).
The emulator is available as a *Linux* and *Windows* Docker container image. It exposes a query endpoint over HTTP that can be used with any client including [Kusto.Explorer](/kusto/tools/kusto-explorer-using?view=azure-data-explorer&preserve-view=true), [Kusto.CLI](/kusto/tools/kusto-cli?view=azure-data-explorer&preserve-view=true), or [Kusto.Data SDKs](/kusto/api/netfx/about-kusto-data?view=azure-data-explorer&preserve-view=true).

![Schematic representation of Kusto emulator architecture.](media/kusto-emulator/kusto-emulator-architecture.png)

Expand All @@ -34,7 +34,7 @@ The main scenarios for the emulator are:

## Kusto emulator vs. free cluster

There are some overlaps between the Kusto emulator and the [free cluster](start-for-free.md) offerings. The following table contrasts the features of the offerings.
There are some overlaps between the Kusto emulator and the [free cluster](start-for-free.md) offerings. The following table contrasts the features of the offerings.

| Item | Kusto emulator | Free cluster |
|--|--|--|
Expand All @@ -49,10 +49,10 @@ There are some overlaps between the Kusto emulator and the [free cluster](start-
| Long term data management | No extent merge capability | Full support |
| Internet requirement | No connectivity required | Cloud service |

When choosing the best option for your use case, you should consider the following:
When choosing the best option for your use case, you should consider:

- **Local development**: The best fit for local development depends on the features you require. For instance, if your scenario requires the use of managed pipelines, the free cluster offering works best. On the other hand, if your scenario requires local development in a disconnected environment, the Kusto emulator would be a better fit.
- **Automated testing**: In general, the Kusto emulator is more suited to automated testing than the free cluster offering. It's faster to provision and doesn't require any Azure principal to set up.
- **Automated testing**: In general, the Kusto emulator is more suited to automated testing than the free cluster offering. It's faster to provision and doesn't require any Azure principal to set up.

[!INCLUDE [kusto-emulator-limitations](includes/kusto-emulator-limitations.md)]

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
ms.topic: include
ms.date: 01/22/2025
---

:::moniker range="azure-data-explorer"
> The examples in this article use publicly available tables in the [help cluster](https://dataexplorer.azure.com/clusters/help/), such as the `StormEvents` table in the *Samples* database.
::: moniker-end
Loading