dashboard architecture

[jbrezmorf]

Current approach

• frontend plots using plotly JS lib
• it uses data provided by backend
• frontend handles events
• it seems that map scatter plot is prepared as a figure layer on the backend side
• adhoc requests, mostly sending full datasets
• custom routes on the backend

Possible solutions

Frontend driven

Pros:

• existing solution
• more responsive
• "edge computing", smaller processing on the backend
  Cons:
• have to handle own frontend-backend protocols
• JB has to learn a TS a bit :-)

Backend driven

Pros:

• simpler, can buils on existing (plotly, dash)
• larger data, but same caching still posible
  Cons:
• less responsive
• need large refactoring of the code

Conclusions:

• want to follow the frontend-driven approach
• need systematicaly designed backend API
• data decimation on backend (in future offline using airflow)

Xpublish solution

• xpublish = basic xarray REST API; based on FastAPI
• xpublish-tiles / xpublish-wms = expose xarray data as map/raster data, dynamic tiling etc.
• xpublish-EDR = exposing zarray data as geospatial datasets, allowing geospatial selections by polygons, etc.
  e.g. could simplify scatter plot of lat-lon points.

Own xpublish extension

• try to use EDR cube endpoint to select reqested coord ranges, but probably does not support slicing
• otherwise: single endpoint, but accept node path arg + variables list + coord slicing -> subarray -> flatten to pandas/polars/arrow parquet/JSON dict of lists -> send

Extension of current map + time series view

• synchronized zoom in time series

Left panel improvements

• dataset overview in the very left panel with node tree as expandable nodes (visually better suggesting the tree structure)
  • put there coords first, then variables as childs of the nodes (datasets)
  • coord info:

    coord<name> (#<len>)
    [<min>  < ... < <max>] for sortable / {<first>, ... , <last>} for unsortable
    type: <type> | unit: <unit representation string> | <doc like icon -> description with markdown support>
    

  - var info: 
   ```
   var <name>  [<coord1>(#<coord1len>), ...]
   [<min>  < ... < <max>] for sortable / {<first>, ... , <last>} for discrete
   type: <type> | unit: <unit representation string> | <doc like icon -> description with markdown support>
   ```
  

Right panel (view) improvements (map-time view)

In future: dropdown menu on top for selecting type of view

Map view on top, multiscale view at bottom, selectors in the middle
selectors

• top: dropdown menus for map view
  variable /coordinate for X axis
  variable /coordinate for Y axis
  variable /coordinate for color (just one for now, at most 4 in future -> subpixels)
  variable /coordinate for point size

  if X,Y are coords -> map plot
  if X or Y is var -> scatter plot over union of X, Y variables coords

• middle universal selectors for coords not in map or multiscale view
  small discrete coord : checkboxes
  large discrete coord : manual list with autocompletion
  sortable : value + range slider
  reduce dataset to given values of these coords

• bottom: dropdown menus for multiscale view
  coordinate for 'time(X)' axis
  variable /coordinate for Y axis
  variable /coordinate for color (dicrete or continuous palete according to the kind of variable)
  variable /coordinate for point size
  variable /coordinate for line type

[jbrezmorf]

HoloViz platform

Integrates existing tools into ecosystem of Python libs for fast development of custom data visualization apps.

Panel

• A way to compose visualization web apps from Python using anywidget platform.
• Unlike Jupyter supports any layout of widgets and combination with JS/TS pacakges.
• Unlike Jupyter does not require single kernel (backend) instance per client, but the bokeh server can handle
  multiple sessions with a single backend.
• Can be combined with custom TS code, but does the frontend - backend synchronization.
• Comes with plenty of ready-to-use components.
• See: VTKSlicer demo combines: GoldenLayout, DynamicMap, basic Panel features

Holoviews

• Unified interface to bokeh, plotly, vega-lite and lagacy matplotlib backend.
  matplotlib - sends rasterized images
  plotly - sends vector graphics primitives
  bokeh - some graphical elements but also the data out of the view - basic panning and zoomin on client side
  vega-lite - full client side rendering, but limited plot options

• provides DynamicMap and reactive parameters to automaticaly update complex plots and link various widget parameters

• Dalayed view updates are simple to implement, letting plots dependent both on widgets as on the updated data, backend can prepare needed datasets offline. Client stays responsive with available data.

Datashader

Seemless replacement of scatterplots for huge datasets. Data are aggregated first into a raster and then rendered as a tiled large image. Can use GPU acceleration or Dask to this end.

Anywidget

Technology that can vastly simplify dashboard implementation is anywidget and use of widgets already developed with it. Holoviz and ather widgets builds on it. It uses
As the interactive DataFrame table Quak. Quak can handle tables with 100M rows of total size of several GB.

Ready to use widgets based on anywidget platfrom

Plotting & Visualization

• Altair (5)
  Declarative statistical visualization library that supports interactive charts in notebooks via widget backends.

• jupyter-scatter (4)
  High-performance WebGL scatter plots capable of interacting with millions of points.

• quak (3)
  Interactive dataset profiling and exploration tool with linked visual summaries.

• Vizarr (4)
  High-performance viewer for large Zarr-based image and volumetric datasets.

• lonboard (4)
  Fast geospatial visualization built on deck.gl and exposed as notebook widgets.

• leafmap (5)
  Interactive geospatial mapping toolkit with rich widget-based controls.

For inspiration

• drawdata (4)
  Interactive canvas for manually sketching datasets directly inside notebooks.

• pyobsplot (3)
  Python bindings for Observable Plot, enabling concise interactive statistical graphics.

• D2 (3)
  Scriptable diagramming language with interactive rendering support in notebook environments.

• HiGlass (5)
  Multiscale genomic visualization system with synchronized tracks and deep zooming.

• itables (5)
  Interactive, sortable, and filterable DataFrame tables for Jupyter environments.

• tldraw (4)
  Whiteboard-style drawing canvas suitable for freeform sketching and collaboration.

• mopaint (2)
  Lightweight pixel-oriented drawing and annotation widget.

• Node editor widgets (2)
  Visual node-based editors for constructing graphs or workflows interactively.

• wigglystuff (2)
  Collection of experimental and demo widgets showcasing modern anywidget patterns.

Zarr -> Duck DB interfaces

Format-agnostic, Xarray / Dask–based Approaches

• xarray-sql
  Provides SQL querying over Xarray.Datasets by flattening multidimensional arrays into tabular form.
  Currently built on dask-sql. Future is unsure, possibly Arrow/DataFusion backend:

SQL -> DataFusion (similar to DuckDB) -> Arrow API -> arrow-zarr -> zarr

• dask-sql
  Executes SQL queries on Dask DataFrames, enabling lazy, distributed evaluation over large datasets. While not multidimensional-native, it is often used as a backend once Xarray data is reshaped into tabular form.

Arrow / DataFusion-Based Backends (DuckDB-adjacent)

• arrow-zarr
  Implements a Zarr reader in Rust that exposes Zarr arrays as Apache Arrow data, enabling Arrow-native query engines (DataFusion, potentially DuckDB) to scan Zarr directly without format conversion.

• Apache DataFusion
  A Rust-based, vectorized SQL query engine built on Apache Arrow. Often used experimentally as a backend for querying scientific datasets once they are exposed via Arrow (e.g., through xarray-sql or arrow-zarr).

Parquet-Based “Raster-as-Table” Approaches

• Raquet (Raster + Parquet)
  A specification for storing raster and multidimensional array data in Parquet, where each row represents a tile or chunk. This allows direct, high-performance querying using DuckDB or other Parquet-native SQL engines.

• Kerchunk + Parquet pipelines (ecosystem pattern)
  Uses Kerchunk to map chunked formats (Zarr, NetCDF, HDF5) into reference-based Parquet or Arrow representations, enabling SQL engines to access large datasets without fully materializing them.

DuckDB Native and Extension-Based Integrations

• DuckDB HDF5 Extension
  Adds a read_hdf5() table function to DuckDB, allowing direct, read-only querying of HDF5 datasets as tables. Demonstrates DuckDB’s extensibility toward scientific array formats.

• DuckDB + GDAL (Spatial Extension)
  Provides limited access to raster/vector data via GDAL, though NetCDF raster support is currently incomplete. Indicates interest but not yet full support for multidimensional scientific formats.