From 2d7c9084100a672d8847b9d2ef34573af8b3ab5a Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Mon, 29 Jun 2026 22:27:35 +0000 Subject: [PATCH 1/5] tutorial: add mvp-1 geodesy notebook 1 --- .../NB1-gnss-discovery_retrieval.ipynb | 602 ++++++++++++++++++ 1 file changed, 602 insertions(+) create mode 100644 tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb diff --git a/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb b/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb new file mode 100644 index 0000000..41e4e27 --- /dev/null +++ b/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb @@ -0,0 +1,602 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "3ef3d1da-4ad2-43d2-835b-cb36a8002baf", + "metadata": {}, + "source": [ + "# GNSS Station Discovery and Retrieval" + ] + }, + { + "cell_type": "markdown", + "id": "00a4248a-322c-42e7-96d2-231740c1af8a", + "metadata": {}, + "source": [ + "**Prerequisites:** Working knowledge of Python and Jupyter notebooks, understanding of GNSS data\n", + "\n", + "**GeoLab compute:** Default Image (4 GB RAM, ~0.5 CPU)" + ] + }, + { + "cell_type": "markdown", + "id": "5ce4783d-27e6-4c2d-93ee-b391b0de5c80", + "metadata": {}, + "source": [ + "## Overview\n", + "\n", + "This notebook demonstrates how to retrieve processed GNSS position time series data from the GAGE web services API provided by EarthScope. You will define a geographic study area, identify available stations, visualize them on an interactive map, and download position time series data to your scratch storage. The output can be used to quickly visualize the GNSS stations available in your Area of Interest (AOI) and download the position time series which can then be used for future workflows.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "f2cdf0e7-2742-41da-a3c6-708b76ba6a07", + "metadata": { + "jp-MarkdownHeadingCollapsed": true + }, + "source": [ + "## Learning Objectives\n", + "\n", + "By the end of this notebook you will be able to:\n", + "\n", + "1. Query the GAGE API to find GNSS stations within a bounding box\n", + "2. Visualize station locations on an interactive map\n", + "3. Retrieve and save processed position time series data for multiple stations\n", + "4. Understand the structure of the GAGE GeoCSV response format" + ] + }, + { + "cell_type": "markdown", + "id": "dc127961-504d-42f6-b5a0-bedbd35665dc", + "metadata": {}, + "source": [ + "## Related Documentation\n", + "\n", + "- [GAGE Web Services Documentation](https://www.unavco.org/data/web-services/documentation/documentation.html#/GNSS47GPS)\n", + "- [GNSS Position Data Documentation](https://www.unavco.org/data/web-services/documentation/gps-position-documentation.html)" + ] + }, + { + "cell_type": "markdown", + "id": "98abba98-c457-45f5-a0af-dbe7ef0ed97e", + "metadata": {}, + "source": [ + "## Setup\n", + "We begin by importing the necessary Python libraries and setting the base URL for all API requests." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "28bde389-1e27-4b6d-a444-0a9078665461", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import shutil\n", + "import requests\n", + "import pandas as pd\n", + "import folium\n", + "import matplotlib.pyplot as plt\n", + "from io import StringIO" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1f9efc80-809a-4157-b848-609648c002e5", + "metadata": {}, + "outputs": [], + "source": [ + "#base URL for GAGE/UNAVCO web service requests.\n", + "BASE_URL = \"https://web-services.unavco.org\"" + ] + }, + { + "cell_type": "markdown", + "id": "00368430-718e-4889-855e-35def3da3af5", + "metadata": {}, + "source": [ + "## Defining Study Area" + ] + }, + { + "cell_type": "markdown", + "id": "776ae7f7-1a99-4918-9b65-c5b66f5cea68", + "metadata": {}, + "source": [ + "We define our study area using a rectangular bounding box that is defined by minimum and maximum latitude and longitude (in degrees). We also define the time range we are interested in using the parameters `START` and `END` in the format `YYYY-MM-DD`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "936d3fc4-95a5-4baa-848c-59e77cb3e129", + "metadata": {}, + "outputs": [], + "source": [ + "MINLAT, MAXLAT = 43, 46\n", + "MINLON, MAXLON = -125, -123\n", + "START, END = \"2017-01-01\", \"2024-01-01\"" + ] + }, + { + "cell_type": "markdown", + "id": "c5c7a794-dd01-4784-a678-caabe9617de1", + "metadata": {}, + "source": [ + "Before retrieving data, we visualize the bounding box on an interactive map to make sure it covers the area we intend. This is a good sanity check to see our coordinates match our intended AOI before making API calls." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eae9ef94-3716-4a99-9b90-d6df34dc5c8d", + "metadata": {}, + "outputs": [], + "source": [ + "def plot_bbox(minlat, maxlat, minlon, maxlon, height=400):\n", + " center_lat = (minlat + maxlat) / 2\n", + " center_lon = (minlon + maxlon) / 2\n", + "\n", + " fig = folium.Figure(height=height)\n", + " m = folium.Map(location=[center_lat, center_lon], zoom_start=7,\n", + " tiles=\"CartoDB positron\")\n", + " m.add_to(fig)\n", + "\n", + " folium.Rectangle(\n", + " bounds=[[minlat, minlon], [maxlat, maxlon]],\n", + " color=\"blue\",\n", + " fill=True,\n", + " fill_opacity=0.1,\n", + " tooltip=f\"Bounding box: ({minlat}, {minlon}) to ({maxlat}, {maxlon})\"\n", + " ).add_to(m)\n", + "\n", + " return fig" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "902eb26a-a826-4fb8-be8a-bc6d2016e2cc", + "metadata": {}, + "outputs": [], + "source": [ + "plot_bbox(MINLAT, MAXLAT, MINLON, MAXLON)" + ] + }, + { + "cell_type": "markdown", + "id": "fcbb166b-7528-4bd9-8a31-1ffb47421ec3", + "metadata": {}, + "source": [ + "## Retrieving Station Metadata" + ] + }, + { + "cell_type": "markdown", + "id": "f01da461-e200-4906-8ff7-3ae5586064b3", + "metadata": {}, + "source": [ + "The GAGE API endpoint `gps/metadata/sites/v1` retrieves site metadata for all GNSS sites that fall within a spatial bounding box defined by north and south latitude, and east and west longitude. The response is a GeoCSV string (a comma separated text format that contains geographic data). Alternatively, we can also fetch the request in a json or XML format.\n", + "\n", + "Let's first fetch the raw response and inspect it before parsing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "816aa057-f568-48d8-9d24-e303f601c588", + "metadata": {}, + "outputs": [], + "source": [ + "def get_stations_in_bbox(minlat, maxlat, minlon, maxlon):\n", + " params = {\n", + " \"minlatitude\": minlat,\n", + " \"maxlatitude\": maxlat,\n", + " \"minlongitude\": minlon,\n", + " \"maxlongitude\": maxlon,\n", + " \"format\": \"csv\"\n", + " }\n", + " r = requests.get(f\"{BASE_URL}/gps/metadata/sites/v1\", params=params) \n", + " r.raise_for_status()\n", + " return r.text" + ] + }, + { + "cell_type": "markdown", + "id": "8d6ace4f-a738-4fa0-809b-050c034ed3f0", + "metadata": {}, + "source": [ + "`requests.get()` returns the entire HTTP response body as a single Python string. We can inspect the raw output as below:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "08af06dd-6b0d-47e4-a6e7-033963ff6b66", + "metadata": {}, + "outputs": [], + "source": [ + "meta_csv = get_stations_in_bbox(MINLAT, MAXLAT, MINLON, MAXLON)\n", + "for line in meta_csv.splitlines()[:3]:\n", + " print(line+\"\\n\")" + ] + }, + { + "cell_type": "markdown", + "id": "b85cca2a-6105-43b2-814a-c2879c751383", + "metadata": {}, + "source": [ + "The first line is the `#fields=` header that encodes column names and types. The following rows after the header row are data. Notice `CORV` appears twice with different receiver types, confirming that stations have multiple session records.\n" + ] + }, + { + "cell_type": "markdown", + "id": "8f5f4fe3-4ee3-44aa-85c5-2ee0e71a5450", + "metadata": {}, + "source": [ + "## Parsing Station Metadata" + ] + }, + { + "cell_type": "markdown", + "id": "fff13051-7723-42e0-9709-e7e6955cf0a0", + "metadata": {}, + "source": [ + "The raw GeoCSV response contains column names encoded in the `#fields=` comment line with type annotations like `[type='string']` that we need to strip out. Each station also appears multiple times in the response — once per equipment configuration (antenna/receiver changes over time). We deduplicate by station ID for getting the station list and optionally filter by our time range of interest." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "576a0863-a9bc-4f82-ba10-1274cd9c9394", + "metadata": {}, + "outputs": [], + "source": [ + "def parse_station_metadata(csv_text, start=None, end=None):\n", + " # extract column names from the #fields= comment line\n", + " fields_line = [l for l in csv_text.splitlines() if l.startswith('#fields=')][0]\n", + " col_names = [f.split('[')[0] for f in fields_line.replace('#fields=', '').split(',')]\n", + "\n", + " # parse data rows (skip all comment lines)\n", + " lines = [l for l in csv_text.splitlines() if not l.startswith('#')]\n", + " df = pd.read_csv(StringIO(\"\\n\".join(lines)), header=None)\n", + " df.columns = col_names\n", + "\n", + " # convert session times to datetime\n", + " df[\"session_start_time\"] = pd.to_datetime(df[\"session_start_time\"])\n", + " df[\"session_stop_time\"] = pd.to_datetime(df[\"session_stop_time\"])\n", + "\n", + " # filter to stations active during our time range\n", + " # a station is included if it has any overlap with [start, end]\n", + " if start:\n", + " df = df[df[\"session_stop_time\"] >= pd.to_datetime(start, utc=True)]\n", + " if end:\n", + " df = df[df[\"session_start_time\"] <= pd.to_datetime(end, utc=True)]\n", + "\n", + " # one row per station\n", + " df = df.drop_duplicates(subset=\"ID\")\n", + "\n", + " return df[[\"ID\", \"latitude\", \"longitude\"]]" + ] + }, + { + "cell_type": "markdown", + "id": "32b41240-6047-4856-b47c-0a244bc4c4d4", + "metadata": {}, + "source": [ + "We now parse the raw response and filter to stations that have data during our time range. The result is one row per station with its ID and coordinates." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "65b95d76-9e5d-4358-80a1-54f537576a92", + "metadata": {}, + "outputs": [], + "source": [ + "stations = parse_station_metadata(meta_csv, start=START, end=END)\n", + "print(f\"Found {len(stations)} stations\")\n", + "stations.reset_index(drop=True).head()" + ] + }, + { + "cell_type": "markdown", + "id": "e1ea2e50-f4fb-44f2-8e63-6d056a5fa33d", + "metadata": {}, + "source": [ + "## Visualizing Stations on a Map" + ] + }, + { + "cell_type": "markdown", + "id": "ddcf39c7-b153-4db2-8a18-4aed87ef9c6f", + "metadata": {}, + "source": [ + "Now that we have our station list, we can add them to the map alongside our bounding box to give us a complete picture of our study area i.e. where the bbox is geographically and which stations fall within it." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a0f8de95-cb87-46f3-9271-c8595e19c4ed", + "metadata": {}, + "outputs": [], + "source": [ + "def plot_stations_map(stations_df, minlat, maxlat, minlon, maxlon, height=500):\n", + " center_lat = (minlat + maxlat) / 2\n", + " center_lon = (minlon + maxlon) / 2\n", + "\n", + " fig = folium.Figure(height=height)\n", + " m = folium.Map(location=[center_lat, center_lon], zoom_start=7,\n", + " tiles=\"CartoDB positron\")\n", + " m.add_to(fig)\n", + "\n", + " # bounding box\n", + " folium.Rectangle(\n", + " bounds=[[minlat, minlon], [maxlat, maxlon]],\n", + " color=\"blue\",\n", + " fill=True,\n", + " fill_opacity=0.1,\n", + " tooltip=f\"Bounding box: ({minlat}, {minlon}) to ({maxlat}, {maxlon})\"\n", + " ).add_to(m)\n", + "\n", + " # station markers\n", + " for _, row in stations_df.iterrows():\n", + " folium.RegularPolygonMarker(\n", + " location=[row[\"latitude\"], row[\"longitude\"]],\n", + " number_of_sides=3,\n", + " radius=5,\n", + " rotation=30,\n", + " color=\"red\",\n", + " fill=True,\n", + " fill_opacity=1,\n", + " popup=row[\"ID\"],\n", + " tooltip=\"Station: \"+row[\"ID\"]\n", + " ).add_to(m)\n", + "\n", + " return fig" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eeb2a430-7877-43c9-b79d-9bff44de03fa", + "metadata": {}, + "outputs": [], + "source": [ + "plot_stations_map(stations, MINLAT, MAXLAT, MINLON, MAXLON)" + ] + }, + { + "cell_type": "markdown", + "id": "7d43dfc4-b2dc-4a71-acc9-77bfb5c3e014", + "metadata": {}, + "source": [ + "You should see red triangle markers within the blue bounding box. Click or hover over a marker to see the station ID. If no stations appear, your bounding box may be too small or your date range too restrictive." + ] + }, + { + "cell_type": "markdown", + "id": "f62e9029-7600-4bb3-857e-542413277761", + "metadata": {}, + "source": [ + "## Retrieving Position Time Series" + ] + }, + { + "cell_type": "markdown", + "id": "55c3afc2-545a-4fa9-b232-395bc8b79abc", + "metadata": {}, + "source": [ + "Now that we have our station list and have verified the locations on the map, we can fetch the processed position time series for each station from the GAGE API endpoint `/gps/data/position/{station}/v3`.\n", + "\n", + "The API returns a response with north, east, and up displacement offsets relative to a reference coordinate at each station. The API parameters we use are:\n", + "\n", + "- **analysisCenter:** GNSS time series position solutions are available from four different analysis centers. CWU (default), NMT, PBO and UNR.\n", + "- **referenceFrame:** The position solutions are available in different reference frames depending on the analysis center. `nam14` (North America fixed) is the default whereas the global frame `igs14` is available from all analysis centers.\n", + "- **report:** controls the output content:\n", + " - `short` - (default) north/east/up offsets and standard deviations only\n", + " - `long` - full source file including cartesian and geodetic coordinates and all associated error estimates and covariances\n", + "- **dataPostProcessing:** post-processing applied after data retrieval:\n", + " - `Uncleaned` (default) — data returned as-is, no post-processing\n", + " - `Cleaned` — offset values for North, East, or Up are set to NULL when the standard deviation exceeds 20mm\n", + "- **format:** Response formats. CSV, JSON or XML.\n", + "\n", + "Data is always fetched fresh from the API and saved to your scratch storage." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "17e7b84c-4ffa-407a-9ed1-5676b7280f65", + "metadata": {}, + "outputs": [], + "source": [ + "SAVE_DIR = os.path.join(os.environ[\"SCRATCH_BUCKET\"], \"gnss_positions\")\n", + "shutil.rmtree(SAVE_DIR) if os.path.exists(SAVE_DIR) else None\n", + "\n", + "def get_position_timeseries(station_code, start=None, end=None):\n", + " params = {\n", + " \"analysisCenter\": \"cwu\",\n", + " \"referenceFrame\": \"nam14\",\n", + " \"report\": \"short\",\n", + " \"dataPostProcessing\": \"Cleaned\",\n", + " \"format\": \"csv\"\n", + " }\n", + " if start:\n", + " params[\"starttime\"] = start\n", + " if end:\n", + " params[\"endtime\"] = end\n", + "\n", + " r = requests.get(f\"{BASE_URL}/gps/data/position/{station_code}/v3\", params=params)\n", + " r.raise_for_status()\n", + " \n", + " os.makedirs(SAVE_DIR, exist_ok=True)\n", + " filepath = os.path.join(SAVE_DIR, f\"{station_code}.csv\")\n", + " with open(filepath, \"w\") as f:\n", + " f.write(r.text)\n", + " # print(f\" Saved to {filepath}\")\n", + " return r.text\n", + "\n", + "\n", + "def parse_position_csv(csv_text):\n", + " lines = [l for l in csv_text.splitlines() if not l.startswith('#')]\n", + " df = pd.read_csv(StringIO(\"\\n\".join(lines)))\n", + " df.columns = df.columns.str.strip()\n", + " df[\"Datetime\"] = pd.to_datetime(df[\"Datetime\"])\n", + " return df" + ] + }, + { + "cell_type": "markdown", + "id": "65bc6ece-a770-49a1-9963-f3fcdd854ef1", + "metadata": {}, + "source": [ + "Before fetching all stations, we test on a single station to verify the response format and confirm our parameters are working correctly." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cea9a987-b648-487e-87bb-b44febbe9597", + "metadata": {}, + "outputs": [], + "source": [ + "# fetch and inspect a single station before running all stations\n", + "code = stations[\"ID\"].iloc[0]\n", + "print(f\"Fetching {code}...\")\n", + "pos_csv = get_position_timeseries(code, start=START, end=END)\n", + "df = parse_position_csv(pos_csv)\n", + "print(f\"{len(df)} epochs\")\n", + "df.head()" + ] + }, + { + "cell_type": "markdown", + "id": "f4568f27-e5a4-424e-991e-b6550f4efff3", + "metadata": {}, + "source": [ + "You should see a DataFrame with columns: `Datetime`, `delta N`, `delta E`, `delta U`, and associated standard deviations. Each row is one daily position estimate. " + ] + }, + { + "cell_type": "markdown", + "id": "e436d329-632a-4258-8253-65bc7bd7201b", + "metadata": {}, + "source": [ + "## Retrieving Data for All Stations" + ] + }, + { + "cell_type": "markdown", + "id": "50e7502c-fcff-46c1-a79f-c9aa72857ecc", + "metadata": {}, + "source": [ + "Now we fetch position time series for all stations. Stations that return a 404 error will be skipped automatically. See the Troubleshooting section for guidance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "38e6a91f-c756-44e8-94b2-f6c85ff97cc8", + "metadata": {}, + "outputs": [], + "source": [ + "for _, row in stations.iterrows():\n", + " code = row[\"ID\"]\n", + " print(f\"Fetching {code}...\")\n", + " try:\n", + " pos_csv = get_position_timeseries(code, start=START, end=END)\n", + " df = parse_position_csv(pos_csv)\n", + " print(f\" {len(df)} epochs\")\n", + " except requests.HTTPError as e:\n", + " print(f\" Skipping {code}: {e}\")" + ] + }, + { + "cell_type": "markdown", + "id": "10ba2f97-b3cd-4d6c-b322-c54c8e1372d0", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "Let's print a summary of what was retrieved to confirm everything looks as expected \n", + "before moving on to analysis." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ed8d0758-0a62-4145-9739-0f0cca2e244d", + "metadata": {}, + "outputs": [], + "source": [ + "successful = []\n", + "failed = []\n", + "\n", + "for _, row in stations.iterrows():\n", + " code = row[\"ID\"]\n", + " filepath = os.path.join(SAVE_DIR, f\"{code}.csv\")\n", + " if os.path.exists(filepath):\n", + " successful.append(code)\n", + " else:\n", + " failed.append(code)\n", + "\n", + "print(f\"Total stations found: {len(stations)}\")\n", + "print(f\"Successfully retrieved: {len(successful)}\")\n", + "print(f\"Failed: {len(failed)}\")\n", + "\n", + "if failed:\n", + " print(f\"\\nFailed stations: {failed}\")\n", + "\n", + "print(f\"\\nData saved to: {SAVE_DIR}\")" + ] + }, + { + "cell_type": "markdown", + "id": "3db5fdc7-54c3-4588-b685-e6aeda88cf6f", + "metadata": {}, + "source": [ + "## Troubleshooting\n", + "\n", + "- If a station returns a 404 error, for example `Skipping station_ID: 404 Client Error` it exists in the metadata but has no processed position solution for your date range. This is normal and the station will be skipped automatically. One solution might be to change the analysis center, for example from `cwu` to `unr` to see if another analysis center has processed the data." + ] + }, + { + "cell_type": "markdown", + "id": "811c5c9a-beb4-4293-8a47-67a809dfb39a", + "metadata": {}, + "source": [ + "## Try It Yourself\n", + "\n", + "- Change `MINLAT`, `MAXLAT`, `MINLON`, `MAXLON` to a different region and re-run\n", + "- Change `START`, `END` to a different time range and re-run\n", + "- Change `report` from `short` to `long`. What additional columns appear?" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 08380b8a33c9216011e849afcb2b1485f591202a Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Sun, 12 Jul 2026 21:46:32 +0000 Subject: [PATCH 2/5] Add notebook 1 from revised roadmap --- .../NB1-access-gnss-via-SDK-checkpoint.ipynb | 614 ++++++++++++++++++ .../NB1-access-gnss-via-SDK.ipynb | 614 ++++++++++++++++++ .../NB1-gnss-discovery_retrieval.ipynb | 602 ----------------- 3 files changed, 1228 insertions(+), 602 deletions(-) create mode 100644 tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb create mode 100644 tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb delete mode 100644 tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb diff --git a/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb new file mode 100644 index 0000000..e3b7dd5 --- /dev/null +++ b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb @@ -0,0 +1,614 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2abe5d1d-4a2f-4c96-a78c-6c3e5f9ac750", + "metadata": {}, + "source": [ + "# Accessing GNSS Observations with the EarthScope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "35320cb4-aed2-4420-8f9c-b3deb365e792", + "metadata": {}, + "source": [ + "**Version:** 1.0 | **Last updated:** 2026-07-09 | **Author:** Eshanta Mishra" + ] + }, + { + "cell_type": "markdown", + "id": "b9b1d1c3-569f-459a-8595-e1ba758e4bc7", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", + "\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to reterive GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "\n", + "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", + "* instantiating the EarthScope SDK client\n", + "* Select data for stations using the station name and time range.\n", + "* Retrieve observations and load into dataframes.\n", + "* Filter (slice) the GNSS data based on different parameters.\n", + "\n", + "---\n", + "\n", + "### Prerequisites\n", + "\n", + "\n", + "Before starting this notebook, you should:\n", + "* [ ] This is an introductory notebook. Being familiar with **Geodesy, GNSS, python dataframes** is recommended but not required.\n", + "\n", + "---\n", + "\n", + "### GeoLab Compute Resources\n", + "\n", + "| Setting | Recommended |\n", + "|---|---|\n", + "| **Image** | GeoLab (default image)|\n", + "| **Server size** | 4 GB RAM, ~0.5 CPUs (default server) |" + ] + }, + { + "cell_type": "markdown", + "id": "811fc68e-1e81-482c-82dd-5efa2a89d715", + "metadata": {}, + "source": [ + "## Learning Objectives\n", + "\n", + "By the end of this notebook, you will be able to:\n", + "\n", + "1. Instantiate EarthScope SDK client\n", + "2. Retreive and filter GNSS data from EarthScope" + ] + }, + { + "cell_type": "markdown", + "id": "1ec3d79a-ad70-4c58-b8fc-287501adb7e9", + "metadata": {}, + "source": [ + "## Relevant Documentation & Resources\n", + "\n", + "* [EarthScope SDK documentation](https://docs.earthscope.org/sdk)" + ] + }, + { + "cell_type": "markdown", + "id": "076d400d-8c88-4113-a28d-91dac8648679", + "metadata": {}, + "source": [ + "## Contents\n", + "\n", + "1. [Basics of GNSS](#id-1-basics-of-gnss)\n", + "2. [Setup & Imports](#id-2-setup-imports)\n", + "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", + "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", + "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", + "6. [Larger than Memory Requests with Query Plan](#id-6-larger-than-memory-requests-with-query-plans)\n", + "7. [Exploration Exercises](#id-7-exploration-exercises)\n", + "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" + ] + }, + { + "cell_type": "markdown", + "id": "1b245a5d-5792-461d-bd51-99e55ab797a0", + "metadata": {}, + "source": [ + "## 1. Basics of GNSS" + ] + }, + { + "cell_type": "markdown", + "id": "9e03f7ee-00b4-4501-bd77-551d49ca52a6", + "metadata": {}, + "source": [ + "A GNSS (Global Navigation Satellite System) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", + "\n", + "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", + "\n", + "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", + "2. **Carrier phase**: It is a more precise distance measurement between the satellite and thr ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", + "3. **SNR**: Signal-to-Noise Ratio. IT is a parameter that tells how strong and clean the received signal is.\n", + "\n", + "**Raw Observations vs. derived products**\n", + "\n", + "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " + ] + }, + { + "cell_type": "markdown", + "id": "cca79b91-0348-4ee2-842e-740595288820", + "metadata": {}, + "source": [ + "## 2. Setup & Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8cd2903c-1b18-4877-a421-f19374f8fcf3", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime as dt\n", + "import polars as pl\n", + "from earthscope_sdk import AsyncEarthScopeClient" + ] + }, + { + "cell_type": "markdown", + "id": "50263265-10c7-44c7-a302-37b658b899c8", + "metadata": {}, + "source": [ + "### Instantiating the Earthscope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "8683b5f9-3ffc-4eb4-a8ec-82c61fa196e7", + "metadata": {}, + "source": [ + "#### Authentication" + ] + }, + { + "cell_type": "markdown", + "id": "52c5e185-8dca-42ff-8a9d-89c23297a761", + "metadata": {}, + "source": [ + "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are always available inside it. So, the client authenticates automatically. This removes the step of logging in or passing tokens manually, as you would do in a non GeoLab Environment." + ] + }, + { + "cell_type": "markdown", + "id": "b0143b85-f435-4ce2-9359-6b5f2b8d3674", + "metadata": {}, + "source": [ + "#### Async client" + ] + }, + { + "cell_type": "markdown", + "id": "421226b8-42bc-4d91-a517-469c29fa6e85", + "metadata": {}, + "source": [ + "`AsyncEarthScopeClient` is the asynchronous client. Async lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "abe833e0-9ca9-48bd-8bfc-749579592f92", + "metadata": {}, + "outputs": [], + "source": [ + "es = AsyncEarthScopeClient()" + ] + }, + { + "cell_type": "markdown", + "id": "aa3e8c77-ceaf-4f6e-be2e-b99b0b7f0125", + "metadata": {}, + "source": [ + "### Configuration\n", + "\n", + "Set your parameters here before running the rest of the notebook. Every subsequent cells read from these variables. So, this is the only place you need to edit to point the notebook at different data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15ecb388-d718-4949-a072-5db6cd398d65", + "metadata": {}, + "outputs": [], + "source": [ + "# Modify these values before running the notebook.\n", + "\n", + "STATION = \"AC60\" # GNSS station (4-character ID)\n", + "SESSION = \"A\" # Session name\n", + "START = dt.datetime(2025, 7, 20, 21) # Query start (UTC)\n", + "END = dt.datetime(2025, 7, 21, 3) # Query end (UTC)" + ] + }, + { + "cell_type": "markdown", + "id": "f2e9e575-a7b1-4dd8-86a9-17f0fd942f15", + "metadata": {}, + "source": [ + "## 3. Retrieve Observations for a single station" + ] + }, + { + "cell_type": "markdown", + "id": "af77148f-64a0-424a-9dc2-2ec34a8f96e8", + "metadata": {}, + "source": [ + "In this step, we will retrieve GNSS observations for a single station and time window as an Apache arrow table through the EarthScope data API.\n", + "\n", + "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", + "\n", + "The Expected output is a single table with one row per satellite, per signal, per epoch" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4eb07e18-b72b-4ed4-99b7-558d935fee31", + "metadata": {}, + "outputs": [], + "source": [ + "# Step: describe the request, then .fetch() to run the query.\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=START,\n", + " end_datetime=END,\n", + " station_name=STATION,\n", + " session_name=SESSION,\n", + ").fetch()\n", + "\n", + "table" + ] + }, + { + "cell_type": "markdown", + "id": "b5d3fab2-0ab0-4231-9681-b6e56a117f9d", + "metadata": {}, + "source": [ + "Convert the Arrow table to a Polars dataframe with `pl.from_arrow(...)`. This is zero-copy, so it is very efficient. Sorting by `timestamp` makes the rows read chronologically." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "82b54655-559f-4b95-8479-a386affe5b91", + "metadata": {}, + "outputs": [], + "source": [ + "df = pl.from_arrow(table).sort(\"timestamp\")\n", + "df" + ] + }, + { + "cell_type": "markdown", + "id": "05678a5b-388b-4d44-8e04-28a8912af92e", + "metadata": {}, + "source": [ + "## 4. Understanding the Observation fields" + ] + }, + { + "cell_type": "markdown", + "id": "6f775aef-3b48-4a78-9d4f-b43b7cd1c143", + "metadata": {}, + "source": [ + "The default data query returns every field - the same information you would find in a RINEX observation file. Let us take a moment to understand the data structure.\n", + "Each row of data is a single measurement (one satellite, one signal, one instant). Inspect the columns, the constellations present, and the range of signal strengths." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1ab79788-7803-4711-aa4f-f977b8e83e7a", + "metadata": {}, + "outputs": [], + "source": [ + "# Inspect structure and coverage\n", + "print(df.schema) # column names and types\n", + "print(\"Constellations:\", df[\"system\"].unique().to_list())\n", + "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())\n", + "\n", + "df.describe() # summary statistics" + ] + }, + { + "cell_type": "markdown", + "id": "c59fb35b-ec60-4879-8a0e-1c6bbe9e4a4a", + "metadata": {}, + "source": [ + "### What each field means\n", + "\n", + "| Column | Type | Meaning |\n", + "|---|---|---|\n", + "| `timestamp` | datetime (UTC) | The epoch of the observation. |\n", + "| `satellite` | int | Satellite number within its constellation (its PRN / slot). Combine with `system` for a globally unique ID. |\n", + "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L`, `5Q` (decoded below). |\n", + "| `range` | float | Pseudorange in meters. It is the apparent satellite to receiver distance. |\n", + "| `phase` | float | Carrier phase in cycles. It is a precise but ambiguous range measurement. |\n", + "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher is cleaner. |\n", + "| `slip` | int | Cycle-slip indicator; `null` when phase tracking was continuous. |\n", + "| `flags` | int | Per-observation status flags (e.g. loss-of-lock). |\n", + "| `fcn` | int | Frequency channel number — used by GLONASS's FDMA signals; `0` otherwise. |\n", + "| `system` | str | Constellation code (see below). |\n", + "| `igs` | str | The station's IGS long name (e.g. `AC6000USA`). |\n", + "\n", + "### Reading the `system` column\n", + "\n", + "| Code | Constellation |\n", + "|---|---|\n", + "| `G` | GPS (USA) |\n", + "| `R` | GLONASS (Russia) |\n", + "| `E` | Galileo (EU) |\n", + "| `C` | BeiDou (China) |\n", + "| `J` | QZSS (Japan) |\n", + "| `I` | NavIC / IRNSS (India) |\n", + "| `S` | SBAS (augmentation) |\n", + "\n", + "### Reading the `obs_code` column\n", + "\n", + "An observation code names a specific signal as **band + tracking attribute**:\n", + "\n", + "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", + "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", + "\n", + "Fpr example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." + ] + }, + { + "cell_type": "markdown", + "id": "7f1cd227-e581-4790-97a7-176bfc846f2a", + "metadata": {}, + "source": [ + "## 5. Requesting Only the Data You Need" + ] + }, + { + "cell_type": "markdown", + "id": "c0a4e066-7b6b-4161-9260-a4a825637f2a", + "metadata": {}, + "source": [ + "When you are working with RINEX files, the files hand you everything, all fields, constellations, satellites etc. Most analyses only need a small slice of these data. Passing filters to `gnss_observations()` in the SDK pushes that selection to the server, so only the data that you asked for is transferred.\n", + "\n", + "Downloading a single signal from a single satellite over two months is drastically cheaper than downloading full RINEX and discarding 99% of it. Less data means faster fetches and smaller memory footprints.\n", + "\n", + "The resulting output is a dataframe containing only the requested stations, constellations, satellites, signals and fields. For example, only `snr` and `range` columns when `field=[\"snr\",\"range\"]`.\n", + "\n", + "Each filter below is optional, accepts a single value or a list, and shrinks the request:\n", + "\n", + "* `station_name`: one name or a list of names\n", + "* `network_name`: request a whole network at once\n", + "* `system`: constellation code(s), e.g. `\"G\"` or `[\"G\", \"R\"]`\n", + "* `satellite`: specific satellite number(s)\n", + "* `obs_code`: specific signal(s), e.g. `[\"1C\", \"2L\"]`\n", + "* `field`: which measurement column(s) to return, e.g. `[\"snr\", \"range\"]`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "949a6db5-f949-43f9-930c-88344765e60b", + "metadata": {}, + "outputs": [], + "source": [ + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " station_name=[\"P717\", \"P453\", \"P146\", \"P147\", \"P041\"],\n", + " session_name=\"A\",\n", + " system=[\"G\", \"R\"],\n", + " obs_code=[\"1C\", \"2L\"],\n", + " satellite=[\"7\", \"21\", \"28\"],\n", + " field=[\"snr\", \"range\"],\n", + ").fetch()\n", + "\n", + "df_sliced = pl.from_arrow(table).sort(\"timestamp\")\n", + "df_sliced" + ] + }, + { + "cell_type": "markdown", + "id": "0cad076e-a951-416e-ad88-96dd7fe1f253", + "metadata": {}, + "source": [ + "> **Check:** The result should contain only the `snr` and `range` measurement columns, no `phase`, no `flags`." + ] + }, + { + "cell_type": "markdown", + "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", + "metadata": {}, + "source": [ + "## 6. Larger than Memory Requests with Query Plan" + ] + }, + { + "cell_type": "markdown", + "id": "81e989fa-baf0-4396-920e-fece9f620f43", + "metadata": {}, + "source": [ + "### What is a query plan?" + ] + }, + { + "cell_type": "markdown", + "id": "7a61354a-548f-4459-976c-89d94b1d5104", + "metadata": {}, + "source": [ + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageble groups such as by day or by station. This allows you to proccess one group at a time.\n", + "\n", + "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", + "\n", + "The expected output is a summary for each group (row count, time span, and stations), rather than one huge dataframe." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1772dd83-792b-4666-a107-392d8522b92f", + "metadata": {}, + "outputs": [], + "source": [ + "# Build a query plan (note: no .fetch())\n", + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "plan = es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " network_name=\"PERM:Alaska\",\n", + " session_name=\"A\",\n", + " system=\"G\",\n", + " field=[\"phase\", \"range\", \"snr\"],\n", + ")\n", + "\n", + "print(plan) # preview the plan's request/group counts before any data is fetched\n", + "\n", + "\n", + "def summarize(table):\n", + " # Print a quick summary of one group's table.\n", + " d = pl.from_arrow(table)\n", + " stations = d[\"igs\"].unique().sort().to_list()\n", + " if len(stations) > 10:\n", + " stations = f\"{len(stations)} stations\"\n", + " print(len(d), d[\"timestamp\"].min(), d[\"timestamp\"].max(), stations)" + ] + }, + { + "cell_type": "markdown", + "id": "6f59fcbd-b77a-4733-82cc-b064b081bc66", + "metadata": {}, + "source": [ + "Printing a plan reports how many API requests and groups it will run before any data moves. This provides the users with a quick way to gauge how large a query is." + ] + }, + { + "cell_type": "markdown", + "id": "ac24a294-a48d-416a-a285-a2abd1f633c5", + "metadata": {}, + "source": [ + "Processing the plan one day at a time. You never hold more than a single day in memory at once." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "46d1af54-622f-4066-9895-cfd595341cdd", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_day():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "02c76245-1e7a-4663-b398-f0472a58eb68", + "metadata": {}, + "source": [ + "Processing the plan one station at a time across the whole window instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "34742255-bd16-4197-b151-1777b4753a1d", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_station():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", + "metadata": {}, + "source": [ + " > check: Each line of output represents one group." + ] + }, + { + "cell_type": "markdown", + "id": "8e88b724-65ec-4ef7-9e19-fb6bbf8c895b", + "metadata": {}, + "source": [ + "### A rough guide for choosing a strategy" + ] + }, + { + "cell_type": "markdown", + "id": "d0673a80-e5e9-4256-8f3f-a48ba2ddafa2", + "metadata": {}, + "source": [ + "* Less than a week: `.fetch()` the whole results at once.\n", + "* Weeks to months, or many stations: iterate with `group_by_day()` (or `group_by_station()`)\n", + "* Months to years: define custom batches with `plan.group_by()`." + ] + }, + { + "cell_type": "markdown", + "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", + "metadata": {}, + "source": [ + "For cutom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [Earthscope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." + ] + }, + { + "cell_type": "markdown", + "id": "c5008800-7811-4fbc-9721-9869da927fc2", + "metadata": {}, + "source": [ + "## 7. Exploration Exercises\n", + "\n", + "Now that you've completed the core workflow, try modifying the parameters below to explore how the results change.\n", + "\n", + "**Try these modifications:**\n", + "\n", + "1. **Change the station:** Set `STATION` in the Configuration section to a different 4-character station ID and re-run Section 3. Does the station report the same constellations?\n", + "\n", + "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"5Q\"`) with `field=\"snr\"`. How much smaller is the result?\n", + "\n", + "3. **Save your results:** Write a fetched dataframe to the scratch directory (*Hint: Use `os.environ[\"SCRATCH_BUCKET\"]`*) as Parquet with `df.write_parquet(...)`, then read it back. Parquet preserves types and is far smaller than CSV." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "edf9b94a-b045-4fcf-be34-f2e2f958f124", + "metadata": {}, + "outputs": [], + "source": [ + "# Exploration cell — use this space to experiment" + ] + }, + { + "cell_type": "markdown", + "id": "1e147f5a-2352-4a16-b28b-3f5c1dbea902", + "metadata": {}, + "source": [ + "## 8. Troubleshooting & Support\n", + "\n", + "### Further Resources\n", + "\n", + "* [EarthScope SDK Documentation](https://docs.earthscope.org/sdk)\n", + "* [SDK GNSS Observation tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", + "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", + "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb new file mode 100644 index 0000000..e3b7dd5 --- /dev/null +++ b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb @@ -0,0 +1,614 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2abe5d1d-4a2f-4c96-a78c-6c3e5f9ac750", + "metadata": {}, + "source": [ + "# Accessing GNSS Observations with the EarthScope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "35320cb4-aed2-4420-8f9c-b3deb365e792", + "metadata": {}, + "source": [ + "**Version:** 1.0 | **Last updated:** 2026-07-09 | **Author:** Eshanta Mishra" + ] + }, + { + "cell_type": "markdown", + "id": "b9b1d1c3-569f-459a-8595-e1ba758e4bc7", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", + "\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to reterive GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "\n", + "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", + "* instantiating the EarthScope SDK client\n", + "* Select data for stations using the station name and time range.\n", + "* Retrieve observations and load into dataframes.\n", + "* Filter (slice) the GNSS data based on different parameters.\n", + "\n", + "---\n", + "\n", + "### Prerequisites\n", + "\n", + "\n", + "Before starting this notebook, you should:\n", + "* [ ] This is an introductory notebook. Being familiar with **Geodesy, GNSS, python dataframes** is recommended but not required.\n", + "\n", + "---\n", + "\n", + "### GeoLab Compute Resources\n", + "\n", + "| Setting | Recommended |\n", + "|---|---|\n", + "| **Image** | GeoLab (default image)|\n", + "| **Server size** | 4 GB RAM, ~0.5 CPUs (default server) |" + ] + }, + { + "cell_type": "markdown", + "id": "811fc68e-1e81-482c-82dd-5efa2a89d715", + "metadata": {}, + "source": [ + "## Learning Objectives\n", + "\n", + "By the end of this notebook, you will be able to:\n", + "\n", + "1. Instantiate EarthScope SDK client\n", + "2. Retreive and filter GNSS data from EarthScope" + ] + }, + { + "cell_type": "markdown", + "id": "1ec3d79a-ad70-4c58-b8fc-287501adb7e9", + "metadata": {}, + "source": [ + "## Relevant Documentation & Resources\n", + "\n", + "* [EarthScope SDK documentation](https://docs.earthscope.org/sdk)" + ] + }, + { + "cell_type": "markdown", + "id": "076d400d-8c88-4113-a28d-91dac8648679", + "metadata": {}, + "source": [ + "## Contents\n", + "\n", + "1. [Basics of GNSS](#id-1-basics-of-gnss)\n", + "2. [Setup & Imports](#id-2-setup-imports)\n", + "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", + "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", + "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", + "6. [Larger than Memory Requests with Query Plan](#id-6-larger-than-memory-requests-with-query-plans)\n", + "7. [Exploration Exercises](#id-7-exploration-exercises)\n", + "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" + ] + }, + { + "cell_type": "markdown", + "id": "1b245a5d-5792-461d-bd51-99e55ab797a0", + "metadata": {}, + "source": [ + "## 1. Basics of GNSS" + ] + }, + { + "cell_type": "markdown", + "id": "9e03f7ee-00b4-4501-bd77-551d49ca52a6", + "metadata": {}, + "source": [ + "A GNSS (Global Navigation Satellite System) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", + "\n", + "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", + "\n", + "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", + "2. **Carrier phase**: It is a more precise distance measurement between the satellite and thr ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", + "3. **SNR**: Signal-to-Noise Ratio. IT is a parameter that tells how strong and clean the received signal is.\n", + "\n", + "**Raw Observations vs. derived products**\n", + "\n", + "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " + ] + }, + { + "cell_type": "markdown", + "id": "cca79b91-0348-4ee2-842e-740595288820", + "metadata": {}, + "source": [ + "## 2. Setup & Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8cd2903c-1b18-4877-a421-f19374f8fcf3", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime as dt\n", + "import polars as pl\n", + "from earthscope_sdk import AsyncEarthScopeClient" + ] + }, + { + "cell_type": "markdown", + "id": "50263265-10c7-44c7-a302-37b658b899c8", + "metadata": {}, + "source": [ + "### Instantiating the Earthscope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "8683b5f9-3ffc-4eb4-a8ec-82c61fa196e7", + "metadata": {}, + "source": [ + "#### Authentication" + ] + }, + { + "cell_type": "markdown", + "id": "52c5e185-8dca-42ff-8a9d-89c23297a761", + "metadata": {}, + "source": [ + "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are always available inside it. So, the client authenticates automatically. This removes the step of logging in or passing tokens manually, as you would do in a non GeoLab Environment." + ] + }, + { + "cell_type": "markdown", + "id": "b0143b85-f435-4ce2-9359-6b5f2b8d3674", + "metadata": {}, + "source": [ + "#### Async client" + ] + }, + { + "cell_type": "markdown", + "id": "421226b8-42bc-4d91-a517-469c29fa6e85", + "metadata": {}, + "source": [ + "`AsyncEarthScopeClient` is the asynchronous client. Async lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "abe833e0-9ca9-48bd-8bfc-749579592f92", + "metadata": {}, + "outputs": [], + "source": [ + "es = AsyncEarthScopeClient()" + ] + }, + { + "cell_type": "markdown", + "id": "aa3e8c77-ceaf-4f6e-be2e-b99b0b7f0125", + "metadata": {}, + "source": [ + "### Configuration\n", + "\n", + "Set your parameters here before running the rest of the notebook. Every subsequent cells read from these variables. So, this is the only place you need to edit to point the notebook at different data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15ecb388-d718-4949-a072-5db6cd398d65", + "metadata": {}, + "outputs": [], + "source": [ + "# Modify these values before running the notebook.\n", + "\n", + "STATION = \"AC60\" # GNSS station (4-character ID)\n", + "SESSION = \"A\" # Session name\n", + "START = dt.datetime(2025, 7, 20, 21) # Query start (UTC)\n", + "END = dt.datetime(2025, 7, 21, 3) # Query end (UTC)" + ] + }, + { + "cell_type": "markdown", + "id": "f2e9e575-a7b1-4dd8-86a9-17f0fd942f15", + "metadata": {}, + "source": [ + "## 3. Retrieve Observations for a single station" + ] + }, + { + "cell_type": "markdown", + "id": "af77148f-64a0-424a-9dc2-2ec34a8f96e8", + "metadata": {}, + "source": [ + "In this step, we will retrieve GNSS observations for a single station and time window as an Apache arrow table through the EarthScope data API.\n", + "\n", + "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", + "\n", + "The Expected output is a single table with one row per satellite, per signal, per epoch" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4eb07e18-b72b-4ed4-99b7-558d935fee31", + "metadata": {}, + "outputs": [], + "source": [ + "# Step: describe the request, then .fetch() to run the query.\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=START,\n", + " end_datetime=END,\n", + " station_name=STATION,\n", + " session_name=SESSION,\n", + ").fetch()\n", + "\n", + "table" + ] + }, + { + "cell_type": "markdown", + "id": "b5d3fab2-0ab0-4231-9681-b6e56a117f9d", + "metadata": {}, + "source": [ + "Convert the Arrow table to a Polars dataframe with `pl.from_arrow(...)`. This is zero-copy, so it is very efficient. Sorting by `timestamp` makes the rows read chronologically." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "82b54655-559f-4b95-8479-a386affe5b91", + "metadata": {}, + "outputs": [], + "source": [ + "df = pl.from_arrow(table).sort(\"timestamp\")\n", + "df" + ] + }, + { + "cell_type": "markdown", + "id": "05678a5b-388b-4d44-8e04-28a8912af92e", + "metadata": {}, + "source": [ + "## 4. Understanding the Observation fields" + ] + }, + { + "cell_type": "markdown", + "id": "6f775aef-3b48-4a78-9d4f-b43b7cd1c143", + "metadata": {}, + "source": [ + "The default data query returns every field - the same information you would find in a RINEX observation file. Let us take a moment to understand the data structure.\n", + "Each row of data is a single measurement (one satellite, one signal, one instant). Inspect the columns, the constellations present, and the range of signal strengths." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1ab79788-7803-4711-aa4f-f977b8e83e7a", + "metadata": {}, + "outputs": [], + "source": [ + "# Inspect structure and coverage\n", + "print(df.schema) # column names and types\n", + "print(\"Constellations:\", df[\"system\"].unique().to_list())\n", + "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())\n", + "\n", + "df.describe() # summary statistics" + ] + }, + { + "cell_type": "markdown", + "id": "c59fb35b-ec60-4879-8a0e-1c6bbe9e4a4a", + "metadata": {}, + "source": [ + "### What each field means\n", + "\n", + "| Column | Type | Meaning |\n", + "|---|---|---|\n", + "| `timestamp` | datetime (UTC) | The epoch of the observation. |\n", + "| `satellite` | int | Satellite number within its constellation (its PRN / slot). Combine with `system` for a globally unique ID. |\n", + "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L`, `5Q` (decoded below). |\n", + "| `range` | float | Pseudorange in meters. It is the apparent satellite to receiver distance. |\n", + "| `phase` | float | Carrier phase in cycles. It is a precise but ambiguous range measurement. |\n", + "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher is cleaner. |\n", + "| `slip` | int | Cycle-slip indicator; `null` when phase tracking was continuous. |\n", + "| `flags` | int | Per-observation status flags (e.g. loss-of-lock). |\n", + "| `fcn` | int | Frequency channel number — used by GLONASS's FDMA signals; `0` otherwise. |\n", + "| `system` | str | Constellation code (see below). |\n", + "| `igs` | str | The station's IGS long name (e.g. `AC6000USA`). |\n", + "\n", + "### Reading the `system` column\n", + "\n", + "| Code | Constellation |\n", + "|---|---|\n", + "| `G` | GPS (USA) |\n", + "| `R` | GLONASS (Russia) |\n", + "| `E` | Galileo (EU) |\n", + "| `C` | BeiDou (China) |\n", + "| `J` | QZSS (Japan) |\n", + "| `I` | NavIC / IRNSS (India) |\n", + "| `S` | SBAS (augmentation) |\n", + "\n", + "### Reading the `obs_code` column\n", + "\n", + "An observation code names a specific signal as **band + tracking attribute**:\n", + "\n", + "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", + "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", + "\n", + "Fpr example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." + ] + }, + { + "cell_type": "markdown", + "id": "7f1cd227-e581-4790-97a7-176bfc846f2a", + "metadata": {}, + "source": [ + "## 5. Requesting Only the Data You Need" + ] + }, + { + "cell_type": "markdown", + "id": "c0a4e066-7b6b-4161-9260-a4a825637f2a", + "metadata": {}, + "source": [ + "When you are working with RINEX files, the files hand you everything, all fields, constellations, satellites etc. Most analyses only need a small slice of these data. Passing filters to `gnss_observations()` in the SDK pushes that selection to the server, so only the data that you asked for is transferred.\n", + "\n", + "Downloading a single signal from a single satellite over two months is drastically cheaper than downloading full RINEX and discarding 99% of it. Less data means faster fetches and smaller memory footprints.\n", + "\n", + "The resulting output is a dataframe containing only the requested stations, constellations, satellites, signals and fields. For example, only `snr` and `range` columns when `field=[\"snr\",\"range\"]`.\n", + "\n", + "Each filter below is optional, accepts a single value or a list, and shrinks the request:\n", + "\n", + "* `station_name`: one name or a list of names\n", + "* `network_name`: request a whole network at once\n", + "* `system`: constellation code(s), e.g. `\"G\"` or `[\"G\", \"R\"]`\n", + "* `satellite`: specific satellite number(s)\n", + "* `obs_code`: specific signal(s), e.g. `[\"1C\", \"2L\"]`\n", + "* `field`: which measurement column(s) to return, e.g. `[\"snr\", \"range\"]`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "949a6db5-f949-43f9-930c-88344765e60b", + "metadata": {}, + "outputs": [], + "source": [ + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " station_name=[\"P717\", \"P453\", \"P146\", \"P147\", \"P041\"],\n", + " session_name=\"A\",\n", + " system=[\"G\", \"R\"],\n", + " obs_code=[\"1C\", \"2L\"],\n", + " satellite=[\"7\", \"21\", \"28\"],\n", + " field=[\"snr\", \"range\"],\n", + ").fetch()\n", + "\n", + "df_sliced = pl.from_arrow(table).sort(\"timestamp\")\n", + "df_sliced" + ] + }, + { + "cell_type": "markdown", + "id": "0cad076e-a951-416e-ad88-96dd7fe1f253", + "metadata": {}, + "source": [ + "> **Check:** The result should contain only the `snr` and `range` measurement columns, no `phase`, no `flags`." + ] + }, + { + "cell_type": "markdown", + "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", + "metadata": {}, + "source": [ + "## 6. Larger than Memory Requests with Query Plan" + ] + }, + { + "cell_type": "markdown", + "id": "81e989fa-baf0-4396-920e-fece9f620f43", + "metadata": {}, + "source": [ + "### What is a query plan?" + ] + }, + { + "cell_type": "markdown", + "id": "7a61354a-548f-4459-976c-89d94b1d5104", + "metadata": {}, + "source": [ + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageble groups such as by day or by station. This allows you to proccess one group at a time.\n", + "\n", + "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", + "\n", + "The expected output is a summary for each group (row count, time span, and stations), rather than one huge dataframe." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1772dd83-792b-4666-a107-392d8522b92f", + "metadata": {}, + "outputs": [], + "source": [ + "# Build a query plan (note: no .fetch())\n", + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "plan = es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " network_name=\"PERM:Alaska\",\n", + " session_name=\"A\",\n", + " system=\"G\",\n", + " field=[\"phase\", \"range\", \"snr\"],\n", + ")\n", + "\n", + "print(plan) # preview the plan's request/group counts before any data is fetched\n", + "\n", + "\n", + "def summarize(table):\n", + " # Print a quick summary of one group's table.\n", + " d = pl.from_arrow(table)\n", + " stations = d[\"igs\"].unique().sort().to_list()\n", + " if len(stations) > 10:\n", + " stations = f\"{len(stations)} stations\"\n", + " print(len(d), d[\"timestamp\"].min(), d[\"timestamp\"].max(), stations)" + ] + }, + { + "cell_type": "markdown", + "id": "6f59fcbd-b77a-4733-82cc-b064b081bc66", + "metadata": {}, + "source": [ + "Printing a plan reports how many API requests and groups it will run before any data moves. This provides the users with a quick way to gauge how large a query is." + ] + }, + { + "cell_type": "markdown", + "id": "ac24a294-a48d-416a-a285-a2abd1f633c5", + "metadata": {}, + "source": [ + "Processing the plan one day at a time. You never hold more than a single day in memory at once." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "46d1af54-622f-4066-9895-cfd595341cdd", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_day():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "02c76245-1e7a-4663-b398-f0472a58eb68", + "metadata": {}, + "source": [ + "Processing the plan one station at a time across the whole window instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "34742255-bd16-4197-b151-1777b4753a1d", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_station():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", + "metadata": {}, + "source": [ + " > check: Each line of output represents one group." + ] + }, + { + "cell_type": "markdown", + "id": "8e88b724-65ec-4ef7-9e19-fb6bbf8c895b", + "metadata": {}, + "source": [ + "### A rough guide for choosing a strategy" + ] + }, + { + "cell_type": "markdown", + "id": "d0673a80-e5e9-4256-8f3f-a48ba2ddafa2", + "metadata": {}, + "source": [ + "* Less than a week: `.fetch()` the whole results at once.\n", + "* Weeks to months, or many stations: iterate with `group_by_day()` (or `group_by_station()`)\n", + "* Months to years: define custom batches with `plan.group_by()`." + ] + }, + { + "cell_type": "markdown", + "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", + "metadata": {}, + "source": [ + "For cutom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [Earthscope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." + ] + }, + { + "cell_type": "markdown", + "id": "c5008800-7811-4fbc-9721-9869da927fc2", + "metadata": {}, + "source": [ + "## 7. Exploration Exercises\n", + "\n", + "Now that you've completed the core workflow, try modifying the parameters below to explore how the results change.\n", + "\n", + "**Try these modifications:**\n", + "\n", + "1. **Change the station:** Set `STATION` in the Configuration section to a different 4-character station ID and re-run Section 3. Does the station report the same constellations?\n", + "\n", + "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"5Q\"`) with `field=\"snr\"`. How much smaller is the result?\n", + "\n", + "3. **Save your results:** Write a fetched dataframe to the scratch directory (*Hint: Use `os.environ[\"SCRATCH_BUCKET\"]`*) as Parquet with `df.write_parquet(...)`, then read it back. Parquet preserves types and is far smaller than CSV." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "edf9b94a-b045-4fcf-be34-f2e2f958f124", + "metadata": {}, + "outputs": [], + "source": [ + "# Exploration cell — use this space to experiment" + ] + }, + { + "cell_type": "markdown", + "id": "1e147f5a-2352-4a16-b28b-3f5c1dbea902", + "metadata": {}, + "source": [ + "## 8. Troubleshooting & Support\n", + "\n", + "### Further Resources\n", + "\n", + "* [EarthScope SDK Documentation](https://docs.earthscope.org/sdk)\n", + "* [SDK GNSS Observation tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", + "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", + "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb b/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb deleted file mode 100644 index 41e4e27..0000000 --- a/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb +++ /dev/null @@ -1,602 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "3ef3d1da-4ad2-43d2-835b-cb36a8002baf", - "metadata": {}, - "source": [ - "# GNSS Station Discovery and Retrieval" - ] - }, - { - "cell_type": "markdown", - "id": "00a4248a-322c-42e7-96d2-231740c1af8a", - "metadata": {}, - "source": [ - "**Prerequisites:** Working knowledge of Python and Jupyter notebooks, understanding of GNSS data\n", - "\n", - "**GeoLab compute:** Default Image (4 GB RAM, ~0.5 CPU)" - ] - }, - { - "cell_type": "markdown", - "id": "5ce4783d-27e6-4c2d-93ee-b391b0de5c80", - "metadata": {}, - "source": [ - "## Overview\n", - "\n", - "This notebook demonstrates how to retrieve processed GNSS position time series data from the GAGE web services API provided by EarthScope. You will define a geographic study area, identify available stations, visualize them on an interactive map, and download position time series data to your scratch storage. The output can be used to quickly visualize the GNSS stations available in your Area of Interest (AOI) and download the position time series which can then be used for future workflows.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "id": "f2cdf0e7-2742-41da-a3c6-708b76ba6a07", - "metadata": { - "jp-MarkdownHeadingCollapsed": true - }, - "source": [ - "## Learning Objectives\n", - "\n", - "By the end of this notebook you will be able to:\n", - "\n", - "1. Query the GAGE API to find GNSS stations within a bounding box\n", - "2. Visualize station locations on an interactive map\n", - "3. Retrieve and save processed position time series data for multiple stations\n", - "4. Understand the structure of the GAGE GeoCSV response format" - ] - }, - { - "cell_type": "markdown", - "id": "dc127961-504d-42f6-b5a0-bedbd35665dc", - "metadata": {}, - "source": [ - "## Related Documentation\n", - "\n", - "- [GAGE Web Services Documentation](https://www.unavco.org/data/web-services/documentation/documentation.html#/GNSS47GPS)\n", - "- [GNSS Position Data Documentation](https://www.unavco.org/data/web-services/documentation/gps-position-documentation.html)" - ] - }, - { - "cell_type": "markdown", - "id": "98abba98-c457-45f5-a0af-dbe7ef0ed97e", - "metadata": {}, - "source": [ - "## Setup\n", - "We begin by importing the necessary Python libraries and setting the base URL for all API requests." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "28bde389-1e27-4b6d-a444-0a9078665461", - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import shutil\n", - "import requests\n", - "import pandas as pd\n", - "import folium\n", - "import matplotlib.pyplot as plt\n", - "from io import StringIO" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1f9efc80-809a-4157-b848-609648c002e5", - "metadata": {}, - "outputs": [], - "source": [ - "#base URL for GAGE/UNAVCO web service requests.\n", - "BASE_URL = \"https://web-services.unavco.org\"" - ] - }, - { - "cell_type": "markdown", - "id": "00368430-718e-4889-855e-35def3da3af5", - "metadata": {}, - "source": [ - "## Defining Study Area" - ] - }, - { - "cell_type": "markdown", - "id": "776ae7f7-1a99-4918-9b65-c5b66f5cea68", - "metadata": {}, - "source": [ - "We define our study area using a rectangular bounding box that is defined by minimum and maximum latitude and longitude (in degrees). We also define the time range we are interested in using the parameters `START` and `END` in the format `YYYY-MM-DD`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "936d3fc4-95a5-4baa-848c-59e77cb3e129", - "metadata": {}, - "outputs": [], - "source": [ - "MINLAT, MAXLAT = 43, 46\n", - "MINLON, MAXLON = -125, -123\n", - "START, END = \"2017-01-01\", \"2024-01-01\"" - ] - }, - { - "cell_type": "markdown", - "id": "c5c7a794-dd01-4784-a678-caabe9617de1", - "metadata": {}, - "source": [ - "Before retrieving data, we visualize the bounding box on an interactive map to make sure it covers the area we intend. This is a good sanity check to see our coordinates match our intended AOI before making API calls." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "eae9ef94-3716-4a99-9b90-d6df34dc5c8d", - "metadata": {}, - "outputs": [], - "source": [ - "def plot_bbox(minlat, maxlat, minlon, maxlon, height=400):\n", - " center_lat = (minlat + maxlat) / 2\n", - " center_lon = (minlon + maxlon) / 2\n", - "\n", - " fig = folium.Figure(height=height)\n", - " m = folium.Map(location=[center_lat, center_lon], zoom_start=7,\n", - " tiles=\"CartoDB positron\")\n", - " m.add_to(fig)\n", - "\n", - " folium.Rectangle(\n", - " bounds=[[minlat, minlon], [maxlat, maxlon]],\n", - " color=\"blue\",\n", - " fill=True,\n", - " fill_opacity=0.1,\n", - " tooltip=f\"Bounding box: ({minlat}, {minlon}) to ({maxlat}, {maxlon})\"\n", - " ).add_to(m)\n", - "\n", - " return fig" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "902eb26a-a826-4fb8-be8a-bc6d2016e2cc", - "metadata": {}, - "outputs": [], - "source": [ - "plot_bbox(MINLAT, MAXLAT, MINLON, MAXLON)" - ] - }, - { - "cell_type": "markdown", - "id": "fcbb166b-7528-4bd9-8a31-1ffb47421ec3", - "metadata": {}, - "source": [ - "## Retrieving Station Metadata" - ] - }, - { - "cell_type": "markdown", - "id": "f01da461-e200-4906-8ff7-3ae5586064b3", - "metadata": {}, - "source": [ - "The GAGE API endpoint `gps/metadata/sites/v1` retrieves site metadata for all GNSS sites that fall within a spatial bounding box defined by north and south latitude, and east and west longitude. The response is a GeoCSV string (a comma separated text format that contains geographic data). Alternatively, we can also fetch the request in a json or XML format.\n", - "\n", - "Let's first fetch the raw response and inspect it before parsing." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "816aa057-f568-48d8-9d24-e303f601c588", - "metadata": {}, - "outputs": [], - "source": [ - "def get_stations_in_bbox(minlat, maxlat, minlon, maxlon):\n", - " params = {\n", - " \"minlatitude\": minlat,\n", - " \"maxlatitude\": maxlat,\n", - " \"minlongitude\": minlon,\n", - " \"maxlongitude\": maxlon,\n", - " \"format\": \"csv\"\n", - " }\n", - " r = requests.get(f\"{BASE_URL}/gps/metadata/sites/v1\", params=params) \n", - " r.raise_for_status()\n", - " return r.text" - ] - }, - { - "cell_type": "markdown", - "id": "8d6ace4f-a738-4fa0-809b-050c034ed3f0", - "metadata": {}, - "source": [ - "`requests.get()` returns the entire HTTP response body as a single Python string. We can inspect the raw output as below:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "08af06dd-6b0d-47e4-a6e7-033963ff6b66", - "metadata": {}, - "outputs": [], - "source": [ - "meta_csv = get_stations_in_bbox(MINLAT, MAXLAT, MINLON, MAXLON)\n", - "for line in meta_csv.splitlines()[:3]:\n", - " print(line+\"\\n\")" - ] - }, - { - "cell_type": "markdown", - "id": "b85cca2a-6105-43b2-814a-c2879c751383", - "metadata": {}, - "source": [ - "The first line is the `#fields=` header that encodes column names and types. The following rows after the header row are data. Notice `CORV` appears twice with different receiver types, confirming that stations have multiple session records.\n" - ] - }, - { - "cell_type": "markdown", - "id": "8f5f4fe3-4ee3-44aa-85c5-2ee0e71a5450", - "metadata": {}, - "source": [ - "## Parsing Station Metadata" - ] - }, - { - "cell_type": "markdown", - "id": "fff13051-7723-42e0-9709-e7e6955cf0a0", - "metadata": {}, - "source": [ - "The raw GeoCSV response contains column names encoded in the `#fields=` comment line with type annotations like `[type='string']` that we need to strip out. Each station also appears multiple times in the response — once per equipment configuration (antenna/receiver changes over time). We deduplicate by station ID for getting the station list and optionally filter by our time range of interest." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "576a0863-a9bc-4f82-ba10-1274cd9c9394", - "metadata": {}, - "outputs": [], - "source": [ - "def parse_station_metadata(csv_text, start=None, end=None):\n", - " # extract column names from the #fields= comment line\n", - " fields_line = [l for l in csv_text.splitlines() if l.startswith('#fields=')][0]\n", - " col_names = [f.split('[')[0] for f in fields_line.replace('#fields=', '').split(',')]\n", - "\n", - " # parse data rows (skip all comment lines)\n", - " lines = [l for l in csv_text.splitlines() if not l.startswith('#')]\n", - " df = pd.read_csv(StringIO(\"\\n\".join(lines)), header=None)\n", - " df.columns = col_names\n", - "\n", - " # convert session times to datetime\n", - " df[\"session_start_time\"] = pd.to_datetime(df[\"session_start_time\"])\n", - " df[\"session_stop_time\"] = pd.to_datetime(df[\"session_stop_time\"])\n", - "\n", - " # filter to stations active during our time range\n", - " # a station is included if it has any overlap with [start, end]\n", - " if start:\n", - " df = df[df[\"session_stop_time\"] >= pd.to_datetime(start, utc=True)]\n", - " if end:\n", - " df = df[df[\"session_start_time\"] <= pd.to_datetime(end, utc=True)]\n", - "\n", - " # one row per station\n", - " df = df.drop_duplicates(subset=\"ID\")\n", - "\n", - " return df[[\"ID\", \"latitude\", \"longitude\"]]" - ] - }, - { - "cell_type": "markdown", - "id": "32b41240-6047-4856-b47c-0a244bc4c4d4", - "metadata": {}, - "source": [ - "We now parse the raw response and filter to stations that have data during our time range. The result is one row per station with its ID and coordinates." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "65b95d76-9e5d-4358-80a1-54f537576a92", - "metadata": {}, - "outputs": [], - "source": [ - "stations = parse_station_metadata(meta_csv, start=START, end=END)\n", - "print(f\"Found {len(stations)} stations\")\n", - "stations.reset_index(drop=True).head()" - ] - }, - { - "cell_type": "markdown", - "id": "e1ea2e50-f4fb-44f2-8e63-6d056a5fa33d", - "metadata": {}, - "source": [ - "## Visualizing Stations on a Map" - ] - }, - { - "cell_type": "markdown", - "id": "ddcf39c7-b153-4db2-8a18-4aed87ef9c6f", - "metadata": {}, - "source": [ - "Now that we have our station list, we can add them to the map alongside our bounding box to give us a complete picture of our study area i.e. where the bbox is geographically and which stations fall within it." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "a0f8de95-cb87-46f3-9271-c8595e19c4ed", - "metadata": {}, - "outputs": [], - "source": [ - "def plot_stations_map(stations_df, minlat, maxlat, minlon, maxlon, height=500):\n", - " center_lat = (minlat + maxlat) / 2\n", - " center_lon = (minlon + maxlon) / 2\n", - "\n", - " fig = folium.Figure(height=height)\n", - " m = folium.Map(location=[center_lat, center_lon], zoom_start=7,\n", - " tiles=\"CartoDB positron\")\n", - " m.add_to(fig)\n", - "\n", - " # bounding box\n", - " folium.Rectangle(\n", - " bounds=[[minlat, minlon], [maxlat, maxlon]],\n", - " color=\"blue\",\n", - " fill=True,\n", - " fill_opacity=0.1,\n", - " tooltip=f\"Bounding box: ({minlat}, {minlon}) to ({maxlat}, {maxlon})\"\n", - " ).add_to(m)\n", - "\n", - " # station markers\n", - " for _, row in stations_df.iterrows():\n", - " folium.RegularPolygonMarker(\n", - " location=[row[\"latitude\"], row[\"longitude\"]],\n", - " number_of_sides=3,\n", - " radius=5,\n", - " rotation=30,\n", - " color=\"red\",\n", - " fill=True,\n", - " fill_opacity=1,\n", - " popup=row[\"ID\"],\n", - " tooltip=\"Station: \"+row[\"ID\"]\n", - " ).add_to(m)\n", - "\n", - " return fig" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "eeb2a430-7877-43c9-b79d-9bff44de03fa", - "metadata": {}, - "outputs": [], - "source": [ - "plot_stations_map(stations, MINLAT, MAXLAT, MINLON, MAXLON)" - ] - }, - { - "cell_type": "markdown", - "id": "7d43dfc4-b2dc-4a71-acc9-77bfb5c3e014", - "metadata": {}, - "source": [ - "You should see red triangle markers within the blue bounding box. Click or hover over a marker to see the station ID. If no stations appear, your bounding box may be too small or your date range too restrictive." - ] - }, - { - "cell_type": "markdown", - "id": "f62e9029-7600-4bb3-857e-542413277761", - "metadata": {}, - "source": [ - "## Retrieving Position Time Series" - ] - }, - { - "cell_type": "markdown", - "id": "55c3afc2-545a-4fa9-b232-395bc8b79abc", - "metadata": {}, - "source": [ - "Now that we have our station list and have verified the locations on the map, we can fetch the processed position time series for each station from the GAGE API endpoint `/gps/data/position/{station}/v3`.\n", - "\n", - "The API returns a response with north, east, and up displacement offsets relative to a reference coordinate at each station. The API parameters we use are:\n", - "\n", - "- **analysisCenter:** GNSS time series position solutions are available from four different analysis centers. CWU (default), NMT, PBO and UNR.\n", - "- **referenceFrame:** The position solutions are available in different reference frames depending on the analysis center. `nam14` (North America fixed) is the default whereas the global frame `igs14` is available from all analysis centers.\n", - "- **report:** controls the output content:\n", - " - `short` - (default) north/east/up offsets and standard deviations only\n", - " - `long` - full source file including cartesian and geodetic coordinates and all associated error estimates and covariances\n", - "- **dataPostProcessing:** post-processing applied after data retrieval:\n", - " - `Uncleaned` (default) — data returned as-is, no post-processing\n", - " - `Cleaned` — offset values for North, East, or Up are set to NULL when the standard deviation exceeds 20mm\n", - "- **format:** Response formats. CSV, JSON or XML.\n", - "\n", - "Data is always fetched fresh from the API and saved to your scratch storage." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "17e7b84c-4ffa-407a-9ed1-5676b7280f65", - "metadata": {}, - "outputs": [], - "source": [ - "SAVE_DIR = os.path.join(os.environ[\"SCRATCH_BUCKET\"], \"gnss_positions\")\n", - "shutil.rmtree(SAVE_DIR) if os.path.exists(SAVE_DIR) else None\n", - "\n", - "def get_position_timeseries(station_code, start=None, end=None):\n", - " params = {\n", - " \"analysisCenter\": \"cwu\",\n", - " \"referenceFrame\": \"nam14\",\n", - " \"report\": \"short\",\n", - " \"dataPostProcessing\": \"Cleaned\",\n", - " \"format\": \"csv\"\n", - " }\n", - " if start:\n", - " params[\"starttime\"] = start\n", - " if end:\n", - " params[\"endtime\"] = end\n", - "\n", - " r = requests.get(f\"{BASE_URL}/gps/data/position/{station_code}/v3\", params=params)\n", - " r.raise_for_status()\n", - " \n", - " os.makedirs(SAVE_DIR, exist_ok=True)\n", - " filepath = os.path.join(SAVE_DIR, f\"{station_code}.csv\")\n", - " with open(filepath, \"w\") as f:\n", - " f.write(r.text)\n", - " # print(f\" Saved to {filepath}\")\n", - " return r.text\n", - "\n", - "\n", - "def parse_position_csv(csv_text):\n", - " lines = [l for l in csv_text.splitlines() if not l.startswith('#')]\n", - " df = pd.read_csv(StringIO(\"\\n\".join(lines)))\n", - " df.columns = df.columns.str.strip()\n", - " df[\"Datetime\"] = pd.to_datetime(df[\"Datetime\"])\n", - " return df" - ] - }, - { - "cell_type": "markdown", - "id": "65bc6ece-a770-49a1-9963-f3fcdd854ef1", - "metadata": {}, - "source": [ - "Before fetching all stations, we test on a single station to verify the response format and confirm our parameters are working correctly." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "cea9a987-b648-487e-87bb-b44febbe9597", - "metadata": {}, - "outputs": [], - "source": [ - "# fetch and inspect a single station before running all stations\n", - "code = stations[\"ID\"].iloc[0]\n", - "print(f\"Fetching {code}...\")\n", - "pos_csv = get_position_timeseries(code, start=START, end=END)\n", - "df = parse_position_csv(pos_csv)\n", - "print(f\"{len(df)} epochs\")\n", - "df.head()" - ] - }, - { - "cell_type": "markdown", - "id": "f4568f27-e5a4-424e-991e-b6550f4efff3", - "metadata": {}, - "source": [ - "You should see a DataFrame with columns: `Datetime`, `delta N`, `delta E`, `delta U`, and associated standard deviations. Each row is one daily position estimate. " - ] - }, - { - "cell_type": "markdown", - "id": "e436d329-632a-4258-8253-65bc7bd7201b", - "metadata": {}, - "source": [ - "## Retrieving Data for All Stations" - ] - }, - { - "cell_type": "markdown", - "id": "50e7502c-fcff-46c1-a79f-c9aa72857ecc", - "metadata": {}, - "source": [ - "Now we fetch position time series for all stations. Stations that return a 404 error will be skipped automatically. See the Troubleshooting section for guidance." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "38e6a91f-c756-44e8-94b2-f6c85ff97cc8", - "metadata": {}, - "outputs": [], - "source": [ - "for _, row in stations.iterrows():\n", - " code = row[\"ID\"]\n", - " print(f\"Fetching {code}...\")\n", - " try:\n", - " pos_csv = get_position_timeseries(code, start=START, end=END)\n", - " df = parse_position_csv(pos_csv)\n", - " print(f\" {len(df)} epochs\")\n", - " except requests.HTTPError as e:\n", - " print(f\" Skipping {code}: {e}\")" - ] - }, - { - "cell_type": "markdown", - "id": "10ba2f97-b3cd-4d6c-b322-c54c8e1372d0", - "metadata": {}, - "source": [ - "## Summary\n", - "\n", - "Let's print a summary of what was retrieved to confirm everything looks as expected \n", - "before moving on to analysis." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "ed8d0758-0a62-4145-9739-0f0cca2e244d", - "metadata": {}, - "outputs": [], - "source": [ - "successful = []\n", - "failed = []\n", - "\n", - "for _, row in stations.iterrows():\n", - " code = row[\"ID\"]\n", - " filepath = os.path.join(SAVE_DIR, f\"{code}.csv\")\n", - " if os.path.exists(filepath):\n", - " successful.append(code)\n", - " else:\n", - " failed.append(code)\n", - "\n", - "print(f\"Total stations found: {len(stations)}\")\n", - "print(f\"Successfully retrieved: {len(successful)}\")\n", - "print(f\"Failed: {len(failed)}\")\n", - "\n", - "if failed:\n", - " print(f\"\\nFailed stations: {failed}\")\n", - "\n", - "print(f\"\\nData saved to: {SAVE_DIR}\")" - ] - }, - { - "cell_type": "markdown", - "id": "3db5fdc7-54c3-4588-b685-e6aeda88cf6f", - "metadata": {}, - "source": [ - "## Troubleshooting\n", - "\n", - "- If a station returns a 404 error, for example `Skipping station_ID: 404 Client Error` it exists in the metadata but has no processed position solution for your date range. This is normal and the station will be skipped automatically. One solution might be to change the analysis center, for example from `cwu` to `unr` to see if another analysis center has processed the data." - ] - }, - { - "cell_type": "markdown", - "id": "811c5c9a-beb4-4293-8a47-67a809dfb39a", - "metadata": {}, - "source": [ - "## Try It Yourself\n", - "\n", - "- Change `MINLAT`, `MAXLAT`, `MINLON`, `MAXLON` to a different region and re-run\n", - "- Change `START`, `END` to a different time range and re-run\n", - "- Change `report` from `short` to `long`. What additional columns appear?" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.13" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} From 66514bf0e1ca232a05fc73712d9f8b7a0c630db1 Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Sun, 12 Jul 2026 21:51:42 +0000 Subject: [PATCH 3/5] Minor grammar fixes --- .../NB1-access-gnss-via-SDK-checkpoint.ipynb | 22 +++++++++---------- .../NB1-access-gnss-via-SDK.ipynb | 22 +++++++++---------- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb index e3b7dd5..1d9c2a6 100644 --- a/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb +++ b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb @@ -25,7 +25,7 @@ "\n", "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", "\n", - "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to reterive GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", "\n", "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", "* instantiating the EarthScope SDK client\n", @@ -61,7 +61,7 @@ "By the end of this notebook, you will be able to:\n", "\n", "1. Instantiate EarthScope SDK client\n", - "2. Retreive and filter GNSS data from EarthScope" + "2. Retrieve and filter GNSS data from EarthScope" ] }, { @@ -86,7 +86,7 @@ "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", - "6. [Larger than Memory Requests with Query Plan](#id-6-larger-than-memory-requests-with-query-plans)\n", + "6. [Larger than Memory Requests with Query Plans](#id-6-larger-than-memory-requests-with-query-plans)\n", "7. [Exploration Exercises](#id-7-exploration-exercises)\n", "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" ] @@ -109,8 +109,8 @@ "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", "\n", "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", - "2. **Carrier phase**: It is a more precise distance measurement between the satellite and thr ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", - "3. **SNR**: Signal-to-Noise Ratio. IT is a parameter that tells how strong and clean the received signal is.\n", + "2. **Carrier phase**: It is a more precise distance measurement between the satellite and the ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", + "3. **SNR**: Signal-to-Noise Ratio. It is a parameter that tells how strong and clean the received signal is.\n", "\n", "**Raw Observations vs. derived products**\n", "\n", @@ -142,7 +142,7 @@ "id": "50263265-10c7-44c7-a302-37b658b899c8", "metadata": {}, "source": [ - "### Instantiating the Earthscope SDK" + "### Instantiating the EarthScope SDK" ] }, { @@ -341,7 +341,7 @@ "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", "\n", - "Fpr example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." + "For example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." ] }, { @@ -411,7 +411,7 @@ "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", "metadata": {}, "source": [ - "## 6. Larger than Memory Requests with Query Plan" + "## 6. Larger than Memory Requests with Query Plans" ] }, { @@ -427,7 +427,7 @@ "id": "7a61354a-548f-4459-976c-89d94b1d5104", "metadata": {}, "source": [ - "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageble groups such as by day or by station. This allows you to proccess one group at a time.\n", + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageable groups such as by day or by station. This allows you to process one group at a time.\n", "\n", "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", "\n", @@ -517,7 +517,7 @@ "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", "metadata": {}, "source": [ - " > check: Each line of output represents one group." + " > **Check**: Each line of output represents one group." ] }, { @@ -543,7 +543,7 @@ "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", "metadata": {}, "source": [ - "For cutom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [Earthscope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." + "For custom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [EarthScope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." ] }, { diff --git a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb index e3b7dd5..1d9c2a6 100644 --- a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb +++ b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb @@ -25,7 +25,7 @@ "\n", "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", "\n", - "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to reterive GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", "\n", "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", "* instantiating the EarthScope SDK client\n", @@ -61,7 +61,7 @@ "By the end of this notebook, you will be able to:\n", "\n", "1. Instantiate EarthScope SDK client\n", - "2. Retreive and filter GNSS data from EarthScope" + "2. Retrieve and filter GNSS data from EarthScope" ] }, { @@ -86,7 +86,7 @@ "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", - "6. [Larger than Memory Requests with Query Plan](#id-6-larger-than-memory-requests-with-query-plans)\n", + "6. [Larger than Memory Requests with Query Plans](#id-6-larger-than-memory-requests-with-query-plans)\n", "7. [Exploration Exercises](#id-7-exploration-exercises)\n", "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" ] @@ -109,8 +109,8 @@ "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", "\n", "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", - "2. **Carrier phase**: It is a more precise distance measurement between the satellite and thr ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", - "3. **SNR**: Signal-to-Noise Ratio. IT is a parameter that tells how strong and clean the received signal is.\n", + "2. **Carrier phase**: It is a more precise distance measurement between the satellite and the ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", + "3. **SNR**: Signal-to-Noise Ratio. It is a parameter that tells how strong and clean the received signal is.\n", "\n", "**Raw Observations vs. derived products**\n", "\n", @@ -142,7 +142,7 @@ "id": "50263265-10c7-44c7-a302-37b658b899c8", "metadata": {}, "source": [ - "### Instantiating the Earthscope SDK" + "### Instantiating the EarthScope SDK" ] }, { @@ -341,7 +341,7 @@ "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", "\n", - "Fpr example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." + "For example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." ] }, { @@ -411,7 +411,7 @@ "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", "metadata": {}, "source": [ - "## 6. Larger than Memory Requests with Query Plan" + "## 6. Larger than Memory Requests with Query Plans" ] }, { @@ -427,7 +427,7 @@ "id": "7a61354a-548f-4459-976c-89d94b1d5104", "metadata": {}, "source": [ - "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageble groups such as by day or by station. This allows you to proccess one group at a time.\n", + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageable groups such as by day or by station. This allows you to process one group at a time.\n", "\n", "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", "\n", @@ -517,7 +517,7 @@ "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", "metadata": {}, "source": [ - " > check: Each line of output represents one group." + " > **Check**: Each line of output represents one group." ] }, { @@ -543,7 +543,7 @@ "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", "metadata": {}, "source": [ - "For cutom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [Earthscope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." + "For custom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [EarthScope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." ] }, { From 5d95d5c1e46a74bc8db284f54ae7bf5d8476c571 Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Mon, 13 Jul 2026 21:36:11 +0000 Subject: [PATCH 4/5] add first geodesy notebook that uses EarthScope SDK --- .../NB1-access-gnss-via-SDK.ipynb | 26 +++++++++---------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb index 1d9c2a6..a69fe75 100644 --- a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb +++ b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb @@ -23,7 +23,7 @@ "source": [ "## Introduction\n", "\n", - "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", + "**What this notebook does:** It instantiates the EarthScope SDK and allows you to retrieve GNSS observations for your station of interest.\n", "\n", "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", "\n", @@ -106,7 +106,7 @@ "source": [ "A GNSS (Global Navigation Satellite System) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", "\n", - "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", + "Each of those recordings is a GNSS observation. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", "\n", "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", "2. **Carrier phase**: It is a more precise distance measurement between the satellite and the ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", @@ -114,7 +114,7 @@ "\n", "**Raw Observations vs. derived products**\n", "\n", - "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " + "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such as a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " ] }, { @@ -225,11 +225,11 @@ "id": "af77148f-64a0-424a-9dc2-2ec34a8f96e8", "metadata": {}, "source": [ - "In this step, we will retrieve GNSS observations for a single station and time window as an Apache arrow table through the EarthScope data API.\n", + "In this step, we will retrieve GNSS observations for a single station and time window as an Apache Arrow table through the EarthScope data API.\n", "\n", "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", "\n", - "The Expected output is a single table with one row per satellite, per signal, per epoch" + "The expected output is a single table with one row per satellite, per signal, per epoch" ] }, { @@ -239,7 +239,7 @@ "metadata": {}, "outputs": [], "source": [ - "# Step: describe the request, then .fetch() to run the query.\n", + "#describe the request, then .fetch() to run the query.\n", "table = await es.data.gnss_observations(\n", " start_datetime=START,\n", " end_datetime=END,\n", @@ -294,11 +294,9 @@ "outputs": [], "source": [ "# Inspect structure and coverage\n", - "print(df.schema) # column names and types\n", - "print(\"Constellations:\", df[\"system\"].unique().to_list())\n", - "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())\n", - "\n", - "df.describe() # summary statistics" + "print(df.schema) \n", + "print(\"Constellations:\", df[\"system\"].unique().sort().to_list())\n", + "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())" ] }, { @@ -312,7 +310,7 @@ "|---|---|---|\n", "| `timestamp` | datetime (UTC) | The epoch of the observation. |\n", "| `satellite` | int | Satellite number within its constellation (its PRN / slot). Combine with `system` for a globally unique ID. |\n", - "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L`, `5Q` (decoded below). |\n", + "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L` (decoded below). |\n", "| `range` | float | Pseudorange in meters. It is the apparent satellite to receiver distance. |\n", "| `phase` | float | Carrier phase in cycles. It is a precise but ambiguous range measurement. |\n", "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher is cleaner. |\n", @@ -453,7 +451,7 @@ " system=\"G\",\n", " field=[\"phase\", \"range\", \"snr\"],\n", ")\n", - "\n", + "await plan.plan()\n", "print(plan) # preview the plan's request/group counts before any data is fetched\n", "\n", "\n", @@ -559,7 +557,7 @@ "\n", "1. **Change the station:** Set `STATION` in the Configuration section to a different 4-character station ID and re-run Section 3. Does the station report the same constellations?\n", "\n", - "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"5Q\"`) with `field=\"snr\"`. How much smaller is the result?\n", + "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"2L\"`) with `field=\"snr\"`. How much smaller is the result?\n", "\n", "3. **Save your results:** Write a fetched dataframe to the scratch directory (*Hint: Use `os.environ[\"SCRATCH_BUCKET\"]`*) as Parquet with `df.write_parquet(...)`, then read it back. Parquet preserves types and is far smaller than CSV." ] From ee44647a7c85696d24906a127af4e612863aa190 Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Fri, 17 Jul 2026 18:35:19 +0000 Subject: [PATCH 5/5] Add notebook 2 - MVP1 geodesy --- .../NB2-instantaneous-PPP-positions.ipynb | 517 ++++++++++++++++++ 1 file changed, 517 insertions(+) create mode 100644 tutorials/mvp-1-geodesy/NB2-instantaneous-PPP-positions.ipynb diff --git a/tutorials/mvp-1-geodesy/NB2-instantaneous-PPP-positions.ipynb b/tutorials/mvp-1-geodesy/NB2-instantaneous-PPP-positions.ipynb new file mode 100644 index 0000000..de0f986 --- /dev/null +++ b/tutorials/mvp-1-geodesy/NB2-instantaneous-PPP-positions.ipynb @@ -0,0 +1,517 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2abe5d1d-4a2f-4c96-a78c-6c3e5f9ac750", + "metadata": {}, + "source": [ + "# Accessing Instantaneous Positions" + ] + }, + { + "cell_type": "markdown", + "id": "35320cb4-aed2-4420-8f9c-b3deb365e792", + "metadata": {}, + "source": [ + "**Version:** 1.0 | **Last updated:** 2026-07-14 | \n", + "\n", + "**Author:** Eshanta Mishra | **Author institution:** EarthScope Consortium\n", + "\n", + "**License:** CC-BY-4.0" + ] + }, + { + "cell_type": "markdown", + "id": "b9b1d1c3-569f-459a-8595-e1ba758e4bc7", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What this notebook does:** The notebook retrieves Instantaneous (high-rate PPP stream) GNSS positions as dataframes for selected stations and time ranges using the EarthScope SDK, and produces visualizations of the position streams and the displacement derived from them.\n", + "\n", + "**Why it is useful:** Instantaneous positions tell us where a station is right now, epoch by epoch. While daily solutions are useful for describing slow tectonic motion, this high-rate stream can capture ground movement that happens in seconds and provides a foundation for studying transient events like earthquakes.\n", + "\n", + "**What you will accomplish:** By the end, you will have retrieved instantaneous positions for one or more stations, inspected and understood data fields that come with the instantaneous positions, derived 2D and 3D displacement magnitudes, and visualized both the position streams and displacement over time.\n", + "\n", + "---\n", + "\n", + "### Prerequisites\n", + "\n", + "Before starting this notebook, you should:\n", + "\n", + "* [ ] Have completed: [Notebook 1 - Accessing GNSS Observations with the EarthScope SDK](NB1-access-gnss-via-SDK.ipynb).\n", + "* [ ] Be familiar with basic Python.\n", + "\n", + "---\n", + "\n", + "### GeoLab Compute Resources\n", + "\n", + "| Setting | Recommended |\n", + "|---|---|\n", + "| **Image** | GeoLab (default image) |\n", + "| **Server size** | 4 GB RAM, ~0.5 CPUs (default server) |" + ] + }, + { + "cell_type": "markdown", + "id": "811fc68e-1e81-482c-82dd-5efa2a89d715", + "metadata": {}, + "source": [ + "## Learning Objectives\n", + "\n", + "By the end of this notebook, you will be able to:\n", + "\n", + "1. Retrieve instantaneous (PPP) positions for one or more stations by station name, time range, facility, and software.\n", + "2. Load the result into a dataframe and interpret every returned field, including position components and their uncertainties.\n", + "3. Derive 2D and 3D displacement magnitudes from the east/north/up components.\n", + "4. Visualize position streams and displacement over time." + ] + }, + { + "cell_type": "markdown", + "id": "1ec3d79a-ad70-4c58-b8fc-287501adb7e9", + "metadata": {}, + "source": [ + "## Relevant Documentation & Resources\n", + "\n", + "* [EarthScope SDK documentation](https://docs.earthscope.org/sdk)\n", + "* [SDK GNSS Observations tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", + "* [Polars User Guide](https://docs.pola.rs)\n", + "* [Altair (plotting)](https://altair-viz.github.io)" + ] + }, + { + "cell_type": "markdown", + "id": "076d400d-8c88-4113-a28d-91dac8648679", + "metadata": {}, + "source": [ + "## Contents\n", + "\n", + "1. [What are Instantaneous Positions?](#id-1-what-are-instantaneous-positions)\n", + "2. [Setup & Imports](#id-2-setup-imports)\n", + "3. [Retrieve Instantaneous Positions](#id-3-retrieve-instantaneous-positions)\n", + "4. [Inspect the Returned Fields](#id-4-inspect-the-returned-fields)\n", + "5. [Derive Displacement](#id-5-derive-displacement)\n", + "6. [Visualize Position Streams & Displacement](#id-6-visualize-position-streams-displacement)\n", + "7. [Exploration Exercises](#id-7-exploration-exercises)\n", + "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" + ] + }, + { + "cell_type": "markdown", + "id": "f6ad6a72-e973-41aa-80f1-7b83114a8c18", + "metadata": {}, + "source": [ + "## 1. What are Instantaneous Positions?" + ] + }, + { + "cell_type": "markdown", + "id": "2b974c85-4624-4a92-94d8-6a097580d88f", + "metadata": {}, + "source": [ + "A GNSS station's position can be estimated in different ways depending on how much data goes into each estimate.\n", + "\n", + "A very common geodetic product is a **daily position time series** which provides one position per station per day, formed by combining a full 24 hours of observations. Averaging over a day suppresses the noise down to millimeter precision. Stacking those daily positions over years yields the velocity fields that reveal slow tectonic motion such as plate drift and interseismic strain at the level of a few millimeters per year. The tradeoff of this precision is the time resolution because one point per day cannot be used to show anything that happens in seconds.\n", + "\n", + "**Instantaneous positions** make the opposite tradeoff. Using Precise Point Positioning (PPP), the station's position is estimated epoch by epoch (once per second in this case). So you can see the current position of the stations continously. Individual estimates are far noisier than daily solutions. However, the high sampling rate makes it possible to detect rapid, transient motion in real time, such as an earthquake displacing a station within seconds. A daily solution would instead combine that motion into a single averaged position.\n", + "\n", + "> **Note:** Derived daily position time series are not yet available through the SDK; this notebook focuses on the instantaneous (high-rate PPP) stream, which is already available." + ] + }, + { + "cell_type": "markdown", + "id": "cca79b91-0348-4ee2-842e-740595288820", + "metadata": {}, + "source": [ + "## 2. Setup & Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8cd2903c-1b18-4877-a421-f19374f8fcf3", + "metadata": {}, + "outputs": [], + "source": [ + "# Standard library imports\n", + "import datetime as dt\n", + "\n", + "# Third-party imports\n", + "import altair as alt\n", + "import polars as pl\n", + "from earthscope_sdk import AsyncEarthScopeClient\n", + "\n", + "# Enable the Rust (vegafusion) backend so Altair can handle larger datasets efficiently\n", + "alt.data_transformers.enable(\"vegafusion\")\n", + "\n", + "es = AsyncEarthScopeClient()" + ] + }, + { + "cell_type": "markdown", + "id": "2448136f-5067-4c1f-98eb-96ea08b06ee4", + "metadata": {}, + "source": [ + "Polars' `.plot` accessor uses Altair under the hood, with vegafusion as its fast backend." + ] + }, + { + "cell_type": "markdown", + "id": "aa3e8c77-ceaf-4f6e-be2e-b99b0b7f0125", + "metadata": {}, + "source": [ + "### Configuration\n", + "\n", + "Set your parameters here before running the rest of the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15ecb388-d718-4949-a072-5db6cd398d65", + "metadata": {}, + "outputs": [], + "source": [ + "# Modify these values before running the notebook.\n", + "\n", + "STATIONS = [\"P146\", \"P147\", \"P148\"] # one or more station IDs\n", + "FACILITY = \"cwu\" # analysis center producing the stream\n", + "SOFTWARE = \"fastlane\" # PPP software producing the stream\n", + "META_FIELDS = [\"geosncl\"] # extra stream-metadata columns to attach\n", + "START = dt.datetime(2026, 6, 4, 10) # query start (UTC)\n", + "END = dt.datetime(2026, 6, 4, 12) # query end (UTC)" + ] + }, + { + "cell_type": "markdown", + "id": "c1c52777-0410-4a6f-b3e6-198cfeebb318", + "metadata": {}, + "source": [ + "## 3. Retrieve Instantaneous Positions\n", + "\n", + "In this step, we will retrieve high-rate PPP position stream for one or more stations, returned as an Apache Arrow table.\n", + "\n", + "As with observations on [Notebook 1](NB1-access-gnss-via-SDK.ipynb), the SDK returns Arrow, which converts into a dataframe with little or no copying. The stream is selected not just by station and time, but also by the processing pipeline that produced it such as the analysis `facility` and the `software`.\n", + "\n", + "Each argument below narrows what you get:\n", + "\n", + "* `station_name`: one station ID or a list\n", + "* `facility`: the analysis center producing the stream (here `\"cwu\"`)\n", + "* `software`: the PPP engine (here `\"fastlane\"`)\n", + "* `meta_fields`: extra stream-metadata columns to attach (here `\"geosncl\"`, the stream identifier)\n", + "\n", + "The expected result is roughly 21,600 rows, i.e. 3 stations x 2 hours x 3600 seconds, since the stream is 1 Hz." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "475dd2e2-9db8-453d-b495-d6afa1417d8b", + "metadata": {}, + "outputs": [], + "source": [ + "# Fetch a 2-hour window of 1 Hz instantaneous positions for the selected stations.\n", + "table = await es.data.gnss_instantaneous_positions(\n", + " start_datetime=START,\n", + " end_datetime=END,\n", + " station_name=STATIONS,\n", + " facility=FACILITY,\n", + " software=SOFTWARE,\n", + " meta_fields=META_FIELDS,\n", + ").fetch()\n", + "\n", + "df = pl.from_arrow(table).sort(\"timestamp\")\n", + "print(f\"{len(df):,} rows\")\n", + "df.head()" + ] + }, + { + "cell_type": "markdown", + "id": "6069a914-e056-472a-a968-516aff9acf58", + "metadata": {}, + "source": [ + "## 4. Inspect the Returned Fields\n", + "\n", + "Get to know the data before analyzing it: which columns came back, how often the stream samples, which stations are present, and where values are missing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "161efa31-b733-4d24-a5de-595b688bf9ae", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"Columns:\", df.columns)\n", + "print(\"Streams:\", df[\"geosncl\"].unique().sort().to_list())" + ] + }, + { + "cell_type": "markdown", + "id": "dbacb342-cda9-4a12-9a58-b47d8e487342", + "metadata": {}, + "source": [ + "### Sampling Interval" + ] + }, + { + "cell_type": "markdown", + "id": "0d4d0465-7f19-43e8-a344-73060dc3ab28", + "metadata": {}, + "source": [ + "Look at the spacing between consecutive epochs for a single stream. It should be 1 seconds." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "92c6c91f-c237-4164-98f9-bae98288a0e0", + "metadata": {}, + "outputs": [], + "source": [ + "one = df.filter(pl.col(\"geosncl\").str.starts_with(STATIONS[0])).sort(\"timestamp\")\n", + "one.select(pl.col(\"timestamp\").diff().alias(\"dt\"))[\"dt\"].drop_nulls().value_counts(sort=True).head()" + ] + }, + { + "cell_type": "markdown", + "id": "c7a92d10-ec53-4cc8-80d2-2909ad699ade", + "metadata": {}, + "source": [ + "### Missing values" + ] + }, + { + "cell_type": "markdown", + "id": "0482145d-6c24-48b4-ab4a-20439d071593", + "metadata": {}, + "source": [ + "Some epochs may lack a horizontal solution, so `east` / `north` can be null even when `up` is present. Count nulls per column. This matters when we derive displacement below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dbfeb030-1f33-4c71-bb81-2cc45ded6ffc", + "metadata": {}, + "outputs": [], + "source": [ + "df.null_count()" + ] + }, + { + "cell_type": "markdown", + "id": "e4ab8121-60a6-4f03-8474-9939434fa14f", + "metadata": {}, + "source": [ + "### What each field means" + ] + }, + { + "cell_type": "markdown", + "id": "d50450ea-b0cb-4398-9f31-30073cb18ab2", + "metadata": {}, + "source": [ + "| Column | Type | Meaning |\n", + "|---|---|---|\n", + "| `timestamp` | datetime (UTC) | Epoch of the position estimate. The stream is 1 Hz, i.e. one row per second, per station. |\n", + "| `east` | float | East displacement from the station's reference position, in meters. |\n", + "| `north` | float | North displacement, in meters. |\n", + "| `up` | float | Vertical (up) displacement, in meters. |\n", + "| `sig_ee` | float | 1-sigma uncertainty on `east`, in meters. |\n", + "| `sig_nn` | float | 1-sigma uncertainty on `north`, in meters. |\n", + "| `sig_uu` | float | 1-sigma uncertainty on `up`, in meters. |\n", + "| `q_channel` | int | Integer-encoded quality and processing status information for each position estimate. |\n", + "| `ingest_latency` | duration | Time between the observation epoch and its ingest by the server. |\n", + "| `processing_delay` | duration | Time taken to process the epoch. |\n", + "| `geosncl` | str | Compound stream identifier: station.network.channel-location (e.g. `P146.PW.LY_.00`). |" + ] + }, + { + "cell_type": "markdown", + "id": "b3f044a4-6ee3-4aba-a4c9-74793839fd72", + "metadata": {}, + "source": [ + "## 5. Derive Displacement\n", + "\n", + "The position components combine into a single displacement magnitude which tells us how far the station sits from its reference position at each epoch. Horizontal (2D) uses east and north; total (3D) adds the vertical:\n", + "\n", + "$$d_{2D} = \\sqrt{east^2 + north^2} \\qquad d_{3D} = \\sqrt{east^2 + north^2 + up^2}$$" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "42de7c31-9908-4286-92fb-4afa03f5f297", + "metadata": {}, + "outputs": [], + "source": [ + "disp = df.with_columns(\n", + " (pl.col(\"east\").pow(2) + pl.col(\"north\").pow(2)).sqrt().alias(\"disp_2d\"),\n", + " (pl.col(\"east\").pow(2) + pl.col(\"north\").pow(2) + pl.col(\"up\").pow(2)).sqrt().alias(\"disp_3d\"),\n", + ")\n", + "\n", + "disp.select([\"timestamp\", \"geosncl\", \"east\", \"north\", \"up\", \"disp_2d\", \"disp_3d\"]).head()" + ] + }, + { + "cell_type": "markdown", + "id": "efe69ab7-0e85-4692-a375-6dfd6a87fa48", + "metadata": {}, + "source": [ + "Note that the square root propagates nulls: if `east` or `north` is missing for an epoch, that epoch's `disp_2d` and `disp_3d` are null too. Drop them before computing statistics if needed using:\n", + "\n", + "```python\n", + "disp_clean = disp.drop_nulls(subset=[\"disp_2d\"])\n", + "```\n", + "\n", + "> **Check:** For a quiet station, horizontal displacement (`disp_2d`) should stay small and roughly steady — a few centimeters of PPP scatter around the reference. A sudden *step* in this value over time is what ground motion (e.g. an earthquake) would look like." + ] + }, + { + "cell_type": "markdown", + "id": "7a66e9a7-c8c7-4b1d-91da-0e4471397815", + "metadata": {}, + "source": [ + "## 6. Visualize Position Streams & Displacement\n", + "\n", + "### The position components over time\n", + "\n", + "Reshape one station's `east` / `north` / `up` into long form and plot them together to see the three streams at once." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "538f9587-fa9f-4e50-a746-b8d2b9c28f25", + "metadata": {}, + "outputs": [], + "source": [ + "one_long = df.filter(pl.col(\"geosncl\").str.starts_with(STATIONS[0])).unpivot(\n", + " [\"east\", \"north\", \"up\"],\n", + " index=\"timestamp\",\n", + " variable_name=\"component\",\n", + " value_name=\"meters\",\n", + ")\n", + "\n", + "one_long.plot.line(x=\"timestamp\", y=\"meters\", color=\"component\").properties(\n", + " width=800, height=300, title=f\"{STATIONS[0]}: position components over time\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "217c22a5-5f14-4bad-81c3-4a64e9bafae6", + "metadata": {}, + "source": [ + "### Displacement over time\n", + "\n", + "Plot the 2D horizontal displacement for every station together. A quiet station traces a roughly flat, noisy band; a step would signal real motion." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c5c5632d-72c5-41fe-8484-37acb86f7091", + "metadata": {}, + "outputs": [], + "source": [ + "disp.plot.line(x=\"timestamp\", y=\"disp_2d\", color=\"geosncl\").properties(\n", + " width=800, height=300, title=\"2D horizontal displacement over time\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "6aa31eda-ecd7-490d-8438-82b45706b732", + "metadata": {}, + "source": [ + "### A note on stream timeliness\n", + "\n", + "The `ingest_latency` and `processing_delay` columns describe how *timely* the stream is, not how accurate the positions are. They are Polars **Duration** types — and Altair can only plot **numeric** axes, so you must convert a Duration to a number (milliseconds) before plotting. This is a common trap." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7d463152-2785-453e-97d4-6f52f5e4e380", + "metadata": {}, + "outputs": [], + "source": [ + "# Altair plots only numeric axes, so convert the Duration column to milliseconds first.\n", + "df.with_columns(\n", + " pl.col(\"ingest_latency\").dt.total_milliseconds().alias(\"ingest_latency_ms\")\n", + ").plot.line(x=\"timestamp\", y=\"ingest_latency_ms\", color=\"geosncl\").properties(\n", + " width=800, height=300, title=\"Ingest latency (ms)\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "c5008800-7811-4fbc-9721-9869da927fc2", + "metadata": {}, + "source": [ + "## 7. Exploration Exercises\n", + "\n", + "Try modifying the parameters to explore how the results change.\n", + "\n", + "1. **Different stations or window:** Change `STATIONS` and the `START`/`END` window in Configuration and re-run. Do all stations return a stream for your window?\n", + "2. **Show the uncertainty:** Plot `up` for one station with a shaded band of +/- `sig_uu` around it (hint: Altair's `mark_area` or `mark_errorband`). How wide is the vertical uncertainty compared to the signal?\n", + "3. **Noisiest station:** Compute the per-station standard deviation of `disp_2d` and identify which station is noisiest." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "edf9b94a-b045-4fcf-be34-f2e2f958f124", + "metadata": {}, + "outputs": [], + "source": [ + "# Exploration cell — use this space to experiment" + ] + }, + { + "cell_type": "markdown", + "id": "1e147f5a-2352-4a16-b28b-3f5c1dbea902", + "metadata": {}, + "source": [ + "## 8. Troubleshooting & Support\n", + "\n", + "### Common Issues\n", + "\n", + "| Error | Likely cause | Fix |\n", + "|---|---|---|\n", + "| Plotting error on a duration column | Altair cannot plot Duration types directly | Convert with `.dt.total_milliseconds()` before plotting |\n", + "| Null `disp_2d` / `disp_3d` values | `east` or `north` was null for that epoch | Use `drop_nulls(subset=[\"disp_2d\"])` before computing statistics |\n", + "\n", + "### Further Resources\n", + "\n", + "* [EarthScope SDK Documentation](https://docs.earthscope.org/sdk)\n", + "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", + "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}