![\"DASK_array\"](\"https://docs.dask.org/en/stable/_images/dask-array.svg\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2b303516",
+ "metadata": {},
+ "source": [
+ "Scientific datasets are often stored as large multidimensional arrays. Examples include seismic waveforms organized by station and time, satellite images, climate grids, and three-dimensional geophysical models. When an array becomes too large to fit comfortably in memory, processing it as a single NumPy array may become difficult or impossible.`dask.array` addresses this problem by dividing a large array into smaller pieces called **chunks**. Each chunk behaves like a NumPy array and can be processed independently. Dask coordinates these chunk-level calculations and combines the partial results to produce the final output.\n",
+ "\n",
+ "In the following example, we will:\n",
+ "\n",
+ "1. Create a two-dimensional Dask Array.\n",
+ "2. Divide the array into manageable chunks.\n",
+ "3. apply a simple mathematical operation.\n",
+ "4. Use `.compute()` to execute the task graph and return the result."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "41c35cb5",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "
| \n",
+ " \n", + " \n", + " | \n", + "
![\"DASK_array\"](\"https://docs.dask.org/en/stable/_images/dask-dataframe.svg\")
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "af083ea2",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " station peak_amplitude snr_db\n",
+ "0 STA01 2.343558 11.035674\n",
+ "1 STA04 0.351853 6.168342\n",
+ "2 STA03 1.562461 5.276518\n",
+ "3 STA02 1.007599 6.092468\n",
+ "4 STA02 6.650617 4.042879\n",
+ "... ... ... ...\n",
+ "199995 STA03 1.580389 4.325777\n",
+ "199996 STA02 0.623621 8.987534\n",
+ "199997 STA01 0.925031 3.225689\n",
+ "199998 STA04 2.994562 4.964203\n",
+ "199999 STA04 2.358059 4.825194\n",
+ "\n",
+ "[200000 rows x 3 columns]"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import pandas as pd\n",
+ "\n",
+ "# Create a small example dataset\n",
+ "# using a random-number generator with a fixed seed\n",
+ "rng = np.random.default_rng(42)\n",
+ "\n",
+ "# Define how many seismic observations will be included.\n",
+ "n_observations = 200_000\n",
+ "\n",
+ "# Create a pandas DataFrame representing measurements collected from four seismic stations\n",
+ "# Each row represents one observation and contains:\n",
+ "# - station: station identifier\n",
+ "# - peak_amplitude: maximum waveform amplitude\n",
+ "# - snr_db: signal-to-noise ratio measured in decibels\n",
+ "station_data = pd.DataFrame({\"station\": rng.choice(\n",
+ " [\"STA01\", \"STA02\", \"STA03\", \"STA04\"],\n",
+ " size=n_observations\n",
+ " ),\n",
+ "\n",
+ " \"peak_amplitude\": rng.lognormal(mean=0.0, sigma=0.8, size=n_observations),\n",
+ "\n",
+ " \"snr_db\": rng.normal(loc=6.0, scale=3.0, size=n_observations)\n",
+ "})\n",
+ "\n",
+ "station_data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3514980c",
+ "metadata": {},
+ "source": [
+ "This cell builds a synthetic dataset of 200,000 seismic observations using a seeded random-number generator, so the results are reproducible. Each row records one measurement from one of four stations, with a `peak_amplitude` drawn from a log-normal distribution and a `snr_db` (signal-to-noise ratio) drawn from a normal distribution. This is an ordinary in-memory **pandas** DataFrame, the starting point that we convert into a partitioned Dask DataFrame in the next cell."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "db70d044",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | station | \n", + "peak_amplitude | \n", + "snr_db | \n", + "
|---|---|---|---|
| 0 | \n", + "STA01 | \n", + "2.343558 | \n", + "11.035674 | \n", + "
| 1 | \n", + "STA04 | \n", + "0.351853 | \n", + "6.168342 | \n", + "
| 2 | \n", + "STA03 | \n", + "1.562461 | \n", + "5.276518 | \n", + "
| 3 | \n", + "STA02 | \n", + "1.007599 | \n", + "6.092468 | \n", + "
| 4 | \n", + "STA02 | \n", + "6.650617 | \n", + "4.042879 | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 199995 | \n", + "STA03 | \n", + "1.580389 | \n", + "4.325777 | \n", + "
| 199996 | \n", + "STA02 | \n", + "0.623621 | \n", + "8.987534 | \n", + "
| 199997 | \n", + "STA01 | \n", + "0.925031 | \n", + "3.225689 | \n", + "
| 199998 | \n", + "STA04 | \n", + "2.994562 | \n", + "4.964203 | \n", + "
| 199999 | \n", + "STA04 | \n", + "2.358059 | \n", + "4.825194 | \n", + "
200000 rows × 3 columns
\n", + "Dask DataFrame Structure:
\n",
+ "| \n", + " | station | \n", + "peak_amplitude | \n", + "snr_db | \n", + "
|---|---|---|---|
| npartitions=8 | \n", + "\n", + " | \n", + " | \n", + " |
| 0 | \n", + "string | \n", + "float64 | \n", + "float64 | \n", + "
| 25000 | \n", + "... | \n", + "... | \n", + "... | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 175000 | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 199999 | \n", + "... | \n", + "... | \n", + "... | \n", + "
Dask Name: frompandas, 1 expression
"
+ ],
+ "text/plain": [
+ "Dask DataFrame Structure:\n",
+ " station peak_amplitude snr_db\n",
+ "npartitions=8 \n",
+ "0 string float64 float64\n",
+ "25000 ... ... ...\n",
+ "... ... ... ...\n",
+ "175000 ... ... ...\n",
+ "199999 ... ... ...\n",
+ "Dask Name: frompandas, 1 expression\n",
+ "Expr=df"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import dask.dataframe as dd\n",
+ "\n",
+ "# Convert the pandas DataFrame into a Dask DataFrame\n",
+ "station_ddf = dd.from_pandas(station_data, npartitions=8)\n",
+ "\n",
+ "station_ddf"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "464fea60",
+ "metadata": {},
+ "source": [
+ "Each partition is a pandas DataFrame containing a portion of the complete dataset. Dask can process these partitions independently,allowing several partitions to be handled in parallel."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f9e0e3af",
+ "metadata": {},
+ "source": [
+ "This example begins with a pandas DataFrame so that it can run quickly in the notebook. In a larger scientific workflow, the complete dataset might not fit in memory. Instead of first loading it with pandas, Dask can construct a partitioned DataFrame directly from a collection of files:\n",
+ ">\n",
+ "> ```python\n",
+ "> station_ddf = dd.read_csv(\"data/station-observations-*.csv\")\n",
+ "> ```\n",
+ ">\n",
+ "or, preferably for many cloud-based analytical workflows:\n",
+ ">\n",
+ "> ```python\n",
+ "> station_ddf = dd.read_parquet(\"data/station-observations/\")\n",
+ "> ```\n",
+ "\n",
+ "Dask initially reads only the dataset metadata. Individual partitions are loaded and processed when a result is requested with `.compute()` or`.persist()`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "94d89f5d",
+ "metadata": {},
+ "source": [
+ "### **4. Dask delayed (building workflows)**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3fa1e663",
+ "metadata": {},
+ "source": [
+ "Dask Array and Dask DataFrame are designed for structured numerical and tabular data. However, many scientific workflows consist of custom Python functions that\n",
+ "retrieve data, apply corrections, calculate measurements, and combine results. These workflows may not fit naturally into an array or DataFrame.\n",
+ "\n",
+ "**Dask Delayed** allows us to parallelize these custom workflows with minimal changes to the original Python code. Wrapping a function with `delayed()` changes when the function is executed:\n",
+ "\n",
+ "- A normal Python function executes immediately and returns its result.\n",
+ "- A delayed function does not execute immediately. It returns a delayed task describing what should be calculated later."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1c7b00e2",
+ "metadata": {},
+ "source": [
+ "Earlier in this notebook, we used Dask Delayed to construct and visualize a simplified seismic-processing workflow. We will now execute that workflow and compare it with the equivalent serial approach."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "aee463e6",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "Serial runtime: 1.42 seconds\n",
+ "\n",
+ "Dask runtime: 0.52 seconds\n"
+ ]
+ }
+ ],
+ "source": [
+ "import time\n",
+ "\n",
+ "# =================================\n",
+ "# ---------- Serial Run -----------\n",
+ "# =================================\n",
+ "\n",
+ "# -- 1. record the starting time --\n",
+ "serial_start = time.perf_counter()\n",
+ "\n",
+ "# -- 2. loading and processing steps --\n",
+ "inventory = load_inventory(stations)\n",
+ "serial_picks = []\n",
+ "for station in stations:\n",
+ "\n",
+ " # Retrieve the waveform for the current station.\n",
+ " waveform = retrieve_waveform(station, inventory)\n",
+ "\n",
+ " # Correct the retrieved waveform.\n",
+ " corrected_waveform = correct_waveform(waveform)\n",
+ "\n",
+ " # Pick the arrival time from the corrected waveform.\n",
+ " arrival_pick = pick_arrival(corrected_waveform)\n",
+ "\n",
+ " # Save the station name and its arrival time.\n",
+ " serial_picks.append(arrival_pick)\n",
+ "\n",
+ "\n",
+ "# Combine the station results into a dictionary.\n",
+ "serial_result = collect_picks(serial_picks)\n",
+ "\n",
+ "# Calculate the total serial execution time.\n",
+ "serial_runtime = time.perf_counter() - serial_start\n",
+ "\n",
+ "print(f\"\\nSerial runtime: {serial_runtime:.2f} seconds\")\n",
+ "\n",
+ "\n",
+ "# =================================\n",
+ "# --------- Parallel Run ----------\n",
+ "# =================================\n",
+ "\n",
+ "# -- 1. Record the starting time --\n",
+ "parallel_start = time.perf_counter()\n",
+ "\n",
+ "# -- 2. Execute processing workflow constructed earlier using delayed() --\n",
+ "parallel_result = workflow.compute()\n",
+ "\n",
+ "# Calculate the total Dask execution time.\n",
+ "parallel_runtime = time.perf_counter() - parallel_start\n",
+ "\n",
+ "print(f\"\\nDask runtime: {parallel_runtime:.2f} seconds\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d07ed078",
+ "metadata": {},
+ "source": [
+ "In the serial workflow, the stations are processed one after another. Python completes waveform retrieval, correction, and arrival picking for one station before beginning the next station. In the Dask workflow, the processing steps within an individual station branch remain sequential because each step depends on the result of the previous one. However, the branches representing different stations are independent and can therefore be assigned to different Dask workers.\n",
+ "\n",
+ "The amount of speedup depends on several factors, including:\n",
+ "\n",
+ "- the number of available workers;\n",
+ "- the number of independent tasks;\n",
+ "- the duration of each task;\n",
+ "- scheduling and communication overhead; and\n",
+ "- whether data must be transferred between workers.\n",
+ "\n",
+ "> **Important:** Dask does not guarantee that every workflow will run faster.\n",
+ "> Small or highly sequential workflows may take longer with Dask because\n",
+ "> scheduling tasks also requires time. Performance should be evaluated using a\n",
+ "> representative portion of the real scientific workflow."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5f4e792a",
+ "metadata": {},
+ "source": [
+ "### **5. DASK futures (dynamic processing)**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0bdd5791",
+ "metadata": {},
+ "source": [
+ "Dask Delayed is useful when we can describe the complete workflow before execution begins. In some scientific applications, however, we may want to submit tasks immediately, monitor their progress, or process individual result as soon as they become available.\n",
+ "\n",
+ "Dask **Futures** provide this more interactive style of execution. A Future is a Python object representing a task that may be waiting, running, completed, or failed on a Dask worker.\n",
+ "\n",
+ "Unlike Delayed tasks, Futures use **eager execution**\n",
+ "\n",
+ "- `delayed()` builds a lazy task graph that waits for `.compute()`.\n",
+ "\n",
+ "whereas\n",
+ "\n",
+ "- `client.submit()` sends an individual task to the scheduler immediately.\n",
+ "- `client.map()` submits the same function for many inputs.\n",
+ "- `client.gather()` retrieves a collection of completed results.\n",
+ "- `as_completed()` returns results one at a time as they finish.\n",
+ "\n",
+ "Futures are especially useful for dynamic workflows, real-time processing, and tasks whose completion times differ. In the following example, we will submit one processing task for each seismic station and inspect the results as they become available."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "5617b341",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Dask DataFrame Structure:
\n",
+ "| \n", + " | station | \n", + "peak_amplitude | \n", + "snr_db | \n", + "
|---|---|---|---|
| npartitions=8 | \n", + "\n", + " | \n", + " | \n", + " |
| 0 | \n", + "string | \n", + "float64 | \n", + "float64 | \n", + "
| 25000 | \n", + "... | \n", + "... | \n", + "... | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 175000 | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 199999 | \n", + "... | \n", + "... | \n", + "... | \n", + "
Dask Name: getitem, 4 expressions
"
+ ],
+ "text/plain": [
+ "Dask DataFrame Structure:\n",
+ " station peak_amplitude snr_db\n",
+ "npartitions=8 \n",
+ "0 string float64 float64\n",
+ "25000 ... ... ...\n",
+ "... ... ... ...\n",
+ "175000 ... ... ...\n",
+ "199999 ... ... ...\n",
+ "Dask Name: getitem, 4 expressions\n",
+ "Expr=Filter(frame=df, predicate=df['snr_db'] >= 3.0)"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Build a lazy intermediate dataset\n",
+ "\n",
+ "quality_controlled = station_ddf[station_ddf[\"snr_db\"] >= 3.0]\n",
+ "\n",
+ "print(type(quality_controlled))\n",
+ "print(f\"Number of partitions:\"f\"{quality_controlled.npartitions}\")\n",
+ "quality_controlled"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9efea38a-b7cd-4754-8702-e48984a3510a",
+ "metadata": {},
+ "source": [
+ "At this point, `quality_controlled` does not contain fully calculated results. It represents the original partitions together with instructions for applying the signal-to-noise filter."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8512dce0-a4bd-4148-a34b-8975f7c37140",
+ "metadata": {},
+ "source": [
+ "#### Use `compute()`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "589c1aa3-4fc7-42d6-8d78-e40d3544a5ab",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Dask DataFrame Structure:
\n",
+ "| \n", + " | station | \n", + "peak_amplitude | \n", + "snr_db | \n", + "
|---|---|---|---|
| npartitions=8 | \n", + "\n", + " | \n", + " | \n", + " |
| 0 | \n", + "string | \n", + "float64 | \n", + "float64 | \n", + "
| 25000 | \n", + "... | \n", + "... | \n", + "... | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 175000 | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 199999 | \n", + "... | \n", + "... | \n", + "... | \n", + "
Dask Name: getitem-ge-getitem, 1 expression
"
+ ],
+ "text/plain": [
+ "Dask DataFrame Structure:\n",
+ " station peak_amplitude snr_db\n",
+ "npartitions=8 \n",
+ "0 string float64 float64\n",
+ "25000 ... ... ...\n",
+ "... ... ... ...\n",
+ "175000 ... ... ...\n",
+ "199999 ... ... ...\n",
+ "Dask Name: getitem-ge-getitem, 1 expression\n",
+ "Expr=FromGraph(448d157)"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from dask.distributed import wait\n",
+ "\n",
+ "# persist() returns immediately with another Dask DataFrame. The\n",
+ "# calculation may still be running in the background.\n",
+ "quality_persisted = quality_controlled.persist()\n",
+ "\n",
+ "# Wait until all persisted partitions have finished calculating.\n",
+ "wait(quality_persisted)\n",
+ "\n",
+ "print(\"The quality-controlled partitions are now stored on the workers.\")\n",
+ "quality_persisted"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "77c81cb2-6fe5-4a81-95bf-a34bfa9a210f",
+ "metadata": {},
+ "source": [
+ "`quality_persisted` remains a Dask DataFrame. Unlike the original lazy\n",
+ "collection, however, its filtered partitions have now been calculated and\n",
+ "stored in distributed worker memory. Downstream calculations can reuse these partitions without repeating the quality-control filter."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dc34df64-3b83-41a0-94cd-7b5bc664f312",
+ "metadata": {},
+ "source": [
+ "| Method | Where is the result stored? | Returned object | Best use |\n",
+ "|---|---|---|---|\n",
+ "| `compute()` | Notebook or client process | NumPy, pandas, or Python object | Retrieving a small final result |\n",
+ "| `persist()` | Distributed worker memory | Dask collection | Reusing an intermediate dataset |\n",
+ "| `dask.compute(a, b)` | Final results return to the notebook | Multiple concrete results | Computing related outputs together |"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": ".conda",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.10"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/tutorials/DASK/dask_scaling.ipynb b/tutorials/DASK/dask_scaling.ipynb
new file mode 100644
index 0000000..9913f99
--- /dev/null
+++ b/tutorials/DASK/dask_scaling.ipynb
@@ -0,0 +1,997 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "9139e97e",
+ "metadata": {},
+ "source": [
+ "# **DASK Scaling**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a00f2483",
+ "metadata": {},
+ "source": [
+ "**Version:** 2.0 | **Last updated:** 2026-08-11\n",
+ "\n",
+ "**Author:** Asif Ashraf | **Author institution:** EarthScope Consortium\n",
+ "\n",
+ "**Maintainer:** EarthScope OnRamp Team | **Maintainer's contact:** help@earthscope.org\n",
+ "\n",
+ "**License:** CC-BY-4.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8d8b0852",
+ "metadata": {},
+ "source": [
+ "## Introduction\n",
+ "\n",
+ "Preparing data from many seismic stations can take time. For each station, we need to download the waveform, clean and resample it, and identify the P-wave arrival. Because each station can be processed independently, we can prepare several waveforms at the same time.\n",
+ "\n",
+ "In this notebook, `dask` distributes these tasks across multiple threads. We then combine the prepared waveforms into a training dataset and use PyTorch to train a small convolutional neural network.\n",
+ "\n",
+ "The goal is to demonstrate a complete workflow: using Dask to prepare seismic data in parallel and then using that data to train a neural network.\n",
+ "\n",
+ "### Learning objectives\n",
+ "\n",
+ "By the end of this notebook, you will be able to:\n",
+ "\n",
+ "- submit independent tasks with Dask `Futures`;\n",
+ "- collect results as tasks finish;\n",
+ "- visualize concurrent execution across threads; and\n",
+ "- pass prepared data from Dask to a neural-network workflow.\n",
+ "\n",
+ "### Contents\n",
+ "\n",
+ "1. [Set up Dask](#1-set-up-dask)\n",
+ "2. [Build the request queue](#2-build-the-request-queue)\n",
+ "3. [Prepare one waveform](#3-prepare-one-waveform)\n",
+ "4. [Submit a Future](#4-submit-a-future)\n",
+ "5. [Stream data in parallel](#5-stream-data-in-parallel)\n",
+ "6. [Build the training dataset](#6-build-the-training-dataset)\n",
+ "7. [Train the CNN](#7-train-the-cnn)\n",
+ "8. [Evaluate predictions](#8-evaluate-predictions)\n",
+ "9. [Conclusion](#9-conclusion)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cc70718f",
+ "metadata": {},
+ "source": [
+ "## 1. Set up Dask\n",
+ "\n",
+ "A Dask `Client` sends tasks to a scheduler. Here, one local Dask process runs four execution threads. Threads share memory and are well suited to this workflow because waveform downloads are I/O-bound and NumPy, ObsPy, and PyTorch perform substantial work outside the Python Global Interpreter Lock."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "a53e2893",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Dask is ready with 4 execution threads.\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ " \n",
+ "
"
+ ],
+ "text/plain": [
+ "\n",
+ " \n",
+ "\n",
+ "
\n",
+ "\n",
+ " \n",
+ "\n",
+ " \n",
+ "
\n",
+ "Client
\n", + "Client-b105565c-96b6-11f1-818c-264305b6bc04
\n", + "| Connection method: Cluster object | \n", + "Cluster type: distributed.LocalCluster | \n", + " \n", + "
| \n", + " Dashboard: /user/google-oauth2%7C112154804904828697217/proxy/8787/status\n", + " | \n", + "\n", + " |
\n",
+ "
\n",
+ "
\n",
+ " \n",
+ "\n",
+ " Cluster Info
\n",
+ " \n",
+ "
\n",
+ " \n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "\n",
+ "
\n",
+ "LocalCluster
\n", + "2859d3fe
\n", + "| \n", + " Dashboard: /user/google-oauth2%7C112154804904828697217/proxy/8787/status\n", + " | \n", + "\n", + " Workers: 1\n", + " | \n", + "
| \n", + " Total threads: 4\n", + " | \n", + "\n", + " Total memory: 3.63 GiB\n", + " | \n", + "
| Status: running | \n", + "Using processes: False | \n", + "
\n",
+ " \n",
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ " Scheduler Info
\n",
+ "
\n",
+ "\n",
+ " \n",
+ "
\n",
+ "\n",
+ " \n",
+ " \n",
+ "
\n",
+ "\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
\n",
+ " Scheduler
\n", + "Scheduler-b67f467b-2857-46ea-b5d0-ca1e8a9ccbc8
\n", + "| \n", + " Comm: inproc://192.168.6.159/396/1\n", + " | \n", + "\n", + " Workers: 0 \n", + " | \n", + "
| \n", + " Dashboard: /user/google-oauth2%7C112154804904828697217/proxy/8787/status\n", + " | \n", + "\n", + " Total threads: 0\n", + " | \n", + "
| \n", + " Started: Just now\n", + " | \n", + "\n", + " Total memory: 0 B\n", + " | \n", + "
\n",
+ " \n",
+ "
\n",
+ "\n",
+ " \n",
+ "
\n",
+ "\n",
+ " Workers
\n",
+ "
\n",
+ "\n",
+ " \n",
+ " \n",
+ " \n",
+ "
\n",
+ " \n",
+ "\n",
+ " \n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ " \n",
+ " Worker: 0
\n",
+ "
\n",
+ " | \n", + " Comm: inproc://192.168.6.159/396/4\n", + " | \n", + "\n", + " Total threads: 4\n", + " | \n", + "
| \n", + " Dashboard: /user/google-oauth2%7C112154804904828697217/proxy/43991/status\n", + " | \n", + "\n", + " Memory: 3.63 GiB\n", + " | \n", + "
| \n", + " Nanny: None\n", + " | \n", + "\n", + " |
| \n", + " Local directory: /tmp/dask-scratch-space/worker-6q9b4b5e\n", + " | \n", + "