QC data filtering and non-standard product loading

docs/myst.yml

@@ -30,6 +30,9 @@ project:
         - file: notebooks/reading_params.ipynb
         - file: notebooks/crossovers.ipynb
         - file: notebooks/bed_picks.ipynb
+        - file: notebooks/qc_demo.ipynb
+        - file: notebooks/nonstandard_data_products.ipynb
+        - file: notebooks/repicking.ipynb
 
 site:
   template: book-theme

docs/notebooks/nonstandard_data_products.ipynb

@@ -0,0 +1,201 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "1772a2d7",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "title: Loading non-standard data products\n",
+    "description: How to use xOPR to load unlisted (non-standard) data products\n",
+    "date: 2026-04-23\n",
+    "---\n",
+    "\n",
+    ":::{warning}\n",
+    "This is an advanced feature and most users can probably ignore it. If you want to load non-standard data products, such as either a custom processing type or individual \"images\" (often corresponding to high and low gain channels), read on.\n",
+    ":::\n",
+    "\n",
+    "By default, xOPR uses a pre-generated STAC catalog of radar data segments and loads certain pre-defined data product types into it, assuming they are available. These include OPR data products such as `CSARP_qlook` (unfocused SAR) and `CSARP_standard` (focused SAR). If you want to work with another data product that is publicly available from the OPR servers but hasn't been indexed by the xOPR STAC catalog (either because you just created it or because it's a non-standard data product that we don't index), you can do this by setting the `allow_unlisted_products=True` flag when loading frames.\n",
+    "\n",
+    "This is a bit of a your-mileage-may-vary feature. xOPR makes a best effort and will probably load most non-standard data products just fine, but a non-standard data product with a completely different format will certainly break things.\n",
+    "\n",
+    "Feel free to reach out if you have a use case that we don't support (or put in a pull request!).\n",
+    "\n",
+    "Keep in mind that any data product must be available on the public side of the OPR data website. If you can't find your desired data product at https://data.cresis.ku.edu/data/rds/, then xOPR can't load it either."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1d5a6e4b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%load_ext autoreload\n",
+    "%autoreload 2\n",
+    "\n",
+    "import numpy as np\n",
+    "import xarray as xr\n",
+    "import geoviews as gv\n",
+    "import geoviews.feature as gf\n",
+    "import cartopy.crs as ccrs\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "import xopr\n",
+    "\n",
+    "import holoviews as hv\n",
+    "import hvplot.xarray\n",
+    "import hvplot.pandas\n",
+    "hvplot.extension('bokeh')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9a567aae",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Establish an OPR session\n",
+    "opr = xopr.OPRConnection(cache_dir=\"radar_cache\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "089dd1ac",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "collection, segment = \"2019_Antarctica_GV\", \"20191105_01\"\n",
+    "\n",
+    "stac_items = opr.query_frames(collections=collection, segment_paths=[segment], max_items=5)\n",
+    "stac_items"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2df228d2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "background_map = gf.ocean.opts(projection=ccrs.SouthPolarStereo(), scale='50m') * gf.coastline.opts(projection=ccrs.SouthPolarStereo(), scale='50m')\n",
+    "background_map * stac_items.to_crs('EPSG:3031').hvplot(aspect='equal')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "affbb5d2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "frames = opr.load_frames(stac_items, data_product=\"CSARP_mvdr\", allow_unlisted_products=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4d075dd2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Inspect an individual frame\n",
+    "frames[0].xopr"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b65d9bf4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "flight_line = xopr.merge_frames(frames)\n",
+    "\n",
+    "stacked = flight_line.resample(slow_time='2s').mean()\n",
+    "stacked.xopr\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(15, 4))\n",
+    "radargram = 10*np.log10(np.abs(stacked['Data']))\n",
+    "radargram.plot.imshow(x='slow_time', cmap='gray', ax=ax)\n",
+    "ax.invert_yaxis()\n",
+    "\n",
+    "ax.set_title(f\"{stacked.attrs['collection']} - {stacked.attrs['segment_path']} - {stacked.attrs['data_product']}\")\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "03ad795e",
+   "metadata": {},
+   "source": [
+    "## Loading individual images\n",
+    "\n",
+    "The `allow_unlisted_products` flag can also be used to enable loading individual \"images\".\n",
+    "\n",
+    ":::{note}\n",
+    "\"Images\" are views of the same segment made from different channels and/or waveforms. They are generally used to capture the full dynamic range between the surface and the bed across different parameters. When you load a default data product, you're actually loading the \"combined\" image, which has been stitched together to build a (hopefully) radiometrically consistent data product.\n",
+    "\n",
+    "Depending on the system, images usually correspond to either a high and low gain channel and/or a set of different transmit waveforms optimized for shallow or deep sounding.\n",
+    ":::"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "aef81f32",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ds_images = {}\n",
+    "for img_idx in range(5):\n",
+    "    try:\n",
+    "        if img_idx == 0:\n",
+    "            img_idx = None\n",
+    "            img_key = \"combined\"\n",
+    "        else:\n",
+    "            img_key = f\"img_{img_idx:02d}\"\n",
+    "        \n",
+    "        ds_images[img_key] = opr.load_frame(stac_items.iloc[0], data_product=\"CSARP_qlook\", image=img_idx, allow_unlisted_products=True)\n",
+    "    except FileNotFoundError:\n",
+    "        break\n",
+    "\n",
+    "slow_time_idx = 100\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(12, 4))\n",
+    "\n",
+    "for img_key, ds_img in ds_images.items():\n",
+    "    img = 10*np.log10(np.abs(ds_img['Data'].isel(slow_time=slow_time_idx)))\n",
+    "    img.plot(x='twtt', ax=ax, label=img_key, alpha=0.6)\n",
+    "\n",
+    "ax.legend()\n",
+    "ax.grid()\n",
+    "ax.set_ylabel(\"return power [dB]\")\n",
+    "plt.show()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "xopr",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}