From 2d7c9084100a672d8847b9d2ef34573af8b3ab5a Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Mon, 29 Jun 2026 22:27:35 +0000 Subject: [PATCH 1/6] tutorial: add mvp-1 geodesy notebook 1 --- .../NB1-gnss-discovery_retrieval.ipynb | 602 ++++++++++++++++++ 1 file changed, 602 insertions(+) create mode 100644 tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb diff --git a/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb b/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb new file mode 100644 index 0000000..41e4e27 --- /dev/null +++ b/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb @@ -0,0 +1,602 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "3ef3d1da-4ad2-43d2-835b-cb36a8002baf", + "metadata": {}, + "source": [ + "# GNSS Station Discovery and Retrieval" + ] + }, + { + "cell_type": "markdown", + "id": "00a4248a-322c-42e7-96d2-231740c1af8a", + "metadata": {}, + "source": [ + "**Prerequisites:** Working knowledge of Python and Jupyter notebooks, understanding of GNSS data\n", + "\n", + "**GeoLab compute:** Default Image (4 GB RAM, ~0.5 CPU)" + ] + }, + { + "cell_type": "markdown", + "id": "5ce4783d-27e6-4c2d-93ee-b391b0de5c80", + "metadata": {}, + "source": [ + "## Overview\n", + "\n", + "This notebook demonstrates how to retrieve processed GNSS position time series data from the GAGE web services API provided by EarthScope. You will define a geographic study area, identify available stations, visualize them on an interactive map, and download position time series data to your scratch storage. The output can be used to quickly visualize the GNSS stations available in your Area of Interest (AOI) and download the position time series which can then be used for future workflows.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "f2cdf0e7-2742-41da-a3c6-708b76ba6a07", + "metadata": { + "jp-MarkdownHeadingCollapsed": true + }, + "source": [ + "## Learning Objectives\n", + "\n", + "By the end of this notebook you will be able to:\n", + "\n", + "1. Query the GAGE API to find GNSS stations within a bounding box\n", + "2. Visualize station locations on an interactive map\n", + "3. Retrieve and save processed position time series data for multiple stations\n", + "4. Understand the structure of the GAGE GeoCSV response format" + ] + }, + { + "cell_type": "markdown", + "id": "dc127961-504d-42f6-b5a0-bedbd35665dc", + "metadata": {}, + "source": [ + "## Related Documentation\n", + "\n", + "- [GAGE Web Services Documentation](https://www.unavco.org/data/web-services/documentation/documentation.html#/GNSS47GPS)\n", + "- [GNSS Position Data Documentation](https://www.unavco.org/data/web-services/documentation/gps-position-documentation.html)" + ] + }, + { + "cell_type": "markdown", + "id": "98abba98-c457-45f5-a0af-dbe7ef0ed97e", + "metadata": {}, + "source": [ + "## Setup\n", + "We begin by importing the necessary Python libraries and setting the base URL for all API requests." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "28bde389-1e27-4b6d-a444-0a9078665461", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import shutil\n", + "import requests\n", + "import pandas as pd\n", + "import folium\n", + "import matplotlib.pyplot as plt\n", + "from io import StringIO" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1f9efc80-809a-4157-b848-609648c002e5", + "metadata": {}, + "outputs": [], + "source": [ + "#base URL for GAGE/UNAVCO web service requests.\n", + "BASE_URL = \"https://web-services.unavco.org\"" + ] + }, + { + "cell_type": "markdown", + "id": "00368430-718e-4889-855e-35def3da3af5", + "metadata": {}, + "source": [ + "## Defining Study Area" + ] + }, + { + "cell_type": "markdown", + "id": "776ae7f7-1a99-4918-9b65-c5b66f5cea68", + "metadata": {}, + "source": [ + "We define our study area using a rectangular bounding box that is defined by minimum and maximum latitude and longitude (in degrees). We also define the time range we are interested in using the parameters `START` and `END` in the format `YYYY-MM-DD`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "936d3fc4-95a5-4baa-848c-59e77cb3e129", + "metadata": {}, + "outputs": [], + "source": [ + "MINLAT, MAXLAT = 43, 46\n", + "MINLON, MAXLON = -125, -123\n", + "START, END = \"2017-01-01\", \"2024-01-01\"" + ] + }, + { + "cell_type": "markdown", + "id": "c5c7a794-dd01-4784-a678-caabe9617de1", + "metadata": {}, + "source": [ + "Before retrieving data, we visualize the bounding box on an interactive map to make sure it covers the area we intend. This is a good sanity check to see our coordinates match our intended AOI before making API calls." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eae9ef94-3716-4a99-9b90-d6df34dc5c8d", + "metadata": {}, + "outputs": [], + "source": [ + "def plot_bbox(minlat, maxlat, minlon, maxlon, height=400):\n", + " center_lat = (minlat + maxlat) / 2\n", + " center_lon = (minlon + maxlon) / 2\n", + "\n", + " fig = folium.Figure(height=height)\n", + " m = folium.Map(location=[center_lat, center_lon], zoom_start=7,\n", + " tiles=\"CartoDB positron\")\n", + " m.add_to(fig)\n", + "\n", + " folium.Rectangle(\n", + " bounds=[[minlat, minlon], [maxlat, maxlon]],\n", + " color=\"blue\",\n", + " fill=True,\n", + " fill_opacity=0.1,\n", + " tooltip=f\"Bounding box: ({minlat}, {minlon}) to ({maxlat}, {maxlon})\"\n", + " ).add_to(m)\n", + "\n", + " return fig" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "902eb26a-a826-4fb8-be8a-bc6d2016e2cc", + "metadata": {}, + "outputs": [], + "source": [ + "plot_bbox(MINLAT, MAXLAT, MINLON, MAXLON)" + ] + }, + { + "cell_type": "markdown", + "id": "fcbb166b-7528-4bd9-8a31-1ffb47421ec3", + "metadata": {}, + "source": [ + "## Retrieving Station Metadata" + ] + }, + { + "cell_type": "markdown", + "id": "f01da461-e200-4906-8ff7-3ae5586064b3", + "metadata": {}, + "source": [ + "The GAGE API endpoint `gps/metadata/sites/v1` retrieves site metadata for all GNSS sites that fall within a spatial bounding box defined by north and south latitude, and east and west longitude. The response is a GeoCSV string (a comma separated text format that contains geographic data). Alternatively, we can also fetch the request in a json or XML format.\n", + "\n", + "Let's first fetch the raw response and inspect it before parsing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "816aa057-f568-48d8-9d24-e303f601c588", + "metadata": {}, + "outputs": [], + "source": [ + "def get_stations_in_bbox(minlat, maxlat, minlon, maxlon):\n", + " params = {\n", + " \"minlatitude\": minlat,\n", + " \"maxlatitude\": maxlat,\n", + " \"minlongitude\": minlon,\n", + " \"maxlongitude\": maxlon,\n", + " \"format\": \"csv\"\n", + " }\n", + " r = requests.get(f\"{BASE_URL}/gps/metadata/sites/v1\", params=params) \n", + " r.raise_for_status()\n", + " return r.text" + ] + }, + { + "cell_type": "markdown", + "id": "8d6ace4f-a738-4fa0-809b-050c034ed3f0", + "metadata": {}, + "source": [ + "`requests.get()` returns the entire HTTP response body as a single Python string. We can inspect the raw output as below:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "08af06dd-6b0d-47e4-a6e7-033963ff6b66", + "metadata": {}, + "outputs": [], + "source": [ + "meta_csv = get_stations_in_bbox(MINLAT, MAXLAT, MINLON, MAXLON)\n", + "for line in meta_csv.splitlines()[:3]:\n", + " print(line+\"\\n\")" + ] + }, + { + "cell_type": "markdown", + "id": "b85cca2a-6105-43b2-814a-c2879c751383", + "metadata": {}, + "source": [ + "The first line is the `#fields=` header that encodes column names and types. The following rows after the header row are data. Notice `CORV` appears twice with different receiver types, confirming that stations have multiple session records.\n" + ] + }, + { + "cell_type": "markdown", + "id": "8f5f4fe3-4ee3-44aa-85c5-2ee0e71a5450", + "metadata": {}, + "source": [ + "## Parsing Station Metadata" + ] + }, + { + "cell_type": "markdown", + "id": "fff13051-7723-42e0-9709-e7e6955cf0a0", + "metadata": {}, + "source": [ + "The raw GeoCSV response contains column names encoded in the `#fields=` comment line with type annotations like `[type='string']` that we need to strip out. Each station also appears multiple times in the response — once per equipment configuration (antenna/receiver changes over time). We deduplicate by station ID for getting the station list and optionally filter by our time range of interest." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "576a0863-a9bc-4f82-ba10-1274cd9c9394", + "metadata": {}, + "outputs": [], + "source": [ + "def parse_station_metadata(csv_text, start=None, end=None):\n", + " # extract column names from the #fields= comment line\n", + " fields_line = [l for l in csv_text.splitlines() if l.startswith('#fields=')][0]\n", + " col_names = [f.split('[')[0] for f in fields_line.replace('#fields=', '').split(',')]\n", + "\n", + " # parse data rows (skip all comment lines)\n", + " lines = [l for l in csv_text.splitlines() if not l.startswith('#')]\n", + " df = pd.read_csv(StringIO(\"\\n\".join(lines)), header=None)\n", + " df.columns = col_names\n", + "\n", + " # convert session times to datetime\n", + " df[\"session_start_time\"] = pd.to_datetime(df[\"session_start_time\"])\n", + " df[\"session_stop_time\"] = pd.to_datetime(df[\"session_stop_time\"])\n", + "\n", + " # filter to stations active during our time range\n", + " # a station is included if it has any overlap with [start, end]\n", + " if start:\n", + " df = df[df[\"session_stop_time\"] >= pd.to_datetime(start, utc=True)]\n", + " if end:\n", + " df = df[df[\"session_start_time\"] <= pd.to_datetime(end, utc=True)]\n", + "\n", + " # one row per station\n", + " df = df.drop_duplicates(subset=\"ID\")\n", + "\n", + " return df[[\"ID\", \"latitude\", \"longitude\"]]" + ] + }, + { + "cell_type": "markdown", + "id": "32b41240-6047-4856-b47c-0a244bc4c4d4", + "metadata": {}, + "source": [ + "We now parse the raw response and filter to stations that have data during our time range. The result is one row per station with its ID and coordinates." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "65b95d76-9e5d-4358-80a1-54f537576a92", + "metadata": {}, + "outputs": [], + "source": [ + "stations = parse_station_metadata(meta_csv, start=START, end=END)\n", + "print(f\"Found {len(stations)} stations\")\n", + "stations.reset_index(drop=True).head()" + ] + }, + { + "cell_type": "markdown", + "id": "e1ea2e50-f4fb-44f2-8e63-6d056a5fa33d", + "metadata": {}, + "source": [ + "## Visualizing Stations on a Map" + ] + }, + { + "cell_type": "markdown", + "id": "ddcf39c7-b153-4db2-8a18-4aed87ef9c6f", + "metadata": {}, + "source": [ + "Now that we have our station list, we can add them to the map alongside our bounding box to give us a complete picture of our study area i.e. where the bbox is geographically and which stations fall within it." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a0f8de95-cb87-46f3-9271-c8595e19c4ed", + "metadata": {}, + "outputs": [], + "source": [ + "def plot_stations_map(stations_df, minlat, maxlat, minlon, maxlon, height=500):\n", + " center_lat = (minlat + maxlat) / 2\n", + " center_lon = (minlon + maxlon) / 2\n", + "\n", + " fig = folium.Figure(height=height)\n", + " m = folium.Map(location=[center_lat, center_lon], zoom_start=7,\n", + " tiles=\"CartoDB positron\")\n", + " m.add_to(fig)\n", + "\n", + " # bounding box\n", + " folium.Rectangle(\n", + " bounds=[[minlat, minlon], [maxlat, maxlon]],\n", + " color=\"blue\",\n", + " fill=True,\n", + " fill_opacity=0.1,\n", + " tooltip=f\"Bounding box: ({minlat}, {minlon}) to ({maxlat}, {maxlon})\"\n", + " ).add_to(m)\n", + "\n", + " # station markers\n", + " for _, row in stations_df.iterrows():\n", + " folium.RegularPolygonMarker(\n", + " location=[row[\"latitude\"], row[\"longitude\"]],\n", + " number_of_sides=3,\n", + " radius=5,\n", + " rotation=30,\n", + " color=\"red\",\n", + " fill=True,\n", + " fill_opacity=1,\n", + " popup=row[\"ID\"],\n", + " tooltip=\"Station: \"+row[\"ID\"]\n", + " ).add_to(m)\n", + "\n", + " return fig" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eeb2a430-7877-43c9-b79d-9bff44de03fa", + "metadata": {}, + "outputs": [], + "source": [ + "plot_stations_map(stations, MINLAT, MAXLAT, MINLON, MAXLON)" + ] + }, + { + "cell_type": "markdown", + "id": "7d43dfc4-b2dc-4a71-acc9-77bfb5c3e014", + "metadata": {}, + "source": [ + "You should see red triangle markers within the blue bounding box. Click or hover over a marker to see the station ID. If no stations appear, your bounding box may be too small or your date range too restrictive." + ] + }, + { + "cell_type": "markdown", + "id": "f62e9029-7600-4bb3-857e-542413277761", + "metadata": {}, + "source": [ + "## Retrieving Position Time Series" + ] + }, + { + "cell_type": "markdown", + "id": "55c3afc2-545a-4fa9-b232-395bc8b79abc", + "metadata": {}, + "source": [ + "Now that we have our station list and have verified the locations on the map, we can fetch the processed position time series for each station from the GAGE API endpoint `/gps/data/position/{station}/v3`.\n", + "\n", + "The API returns a response with north, east, and up displacement offsets relative to a reference coordinate at each station. The API parameters we use are:\n", + "\n", + "- **analysisCenter:** GNSS time series position solutions are available from four different analysis centers. CWU (default), NMT, PBO and UNR.\n", + "- **referenceFrame:** The position solutions are available in different reference frames depending on the analysis center. `nam14` (North America fixed) is the default whereas the global frame `igs14` is available from all analysis centers.\n", + "- **report:** controls the output content:\n", + " - `short` - (default) north/east/up offsets and standard deviations only\n", + " - `long` - full source file including cartesian and geodetic coordinates and all associated error estimates and covariances\n", + "- **dataPostProcessing:** post-processing applied after data retrieval:\n", + " - `Uncleaned` (default) — data returned as-is, no post-processing\n", + " - `Cleaned` — offset values for North, East, or Up are set to NULL when the standard deviation exceeds 20mm\n", + "- **format:** Response formats. CSV, JSON or XML.\n", + "\n", + "Data is always fetched fresh from the API and saved to your scratch storage." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "17e7b84c-4ffa-407a-9ed1-5676b7280f65", + "metadata": {}, + "outputs": [], + "source": [ + "SAVE_DIR = os.path.join(os.environ[\"SCRATCH_BUCKET\"], \"gnss_positions\")\n", + "shutil.rmtree(SAVE_DIR) if os.path.exists(SAVE_DIR) else None\n", + "\n", + "def get_position_timeseries(station_code, start=None, end=None):\n", + " params = {\n", + " \"analysisCenter\": \"cwu\",\n", + " \"referenceFrame\": \"nam14\",\n", + " \"report\": \"short\",\n", + " \"dataPostProcessing\": \"Cleaned\",\n", + " \"format\": \"csv\"\n", + " }\n", + " if start:\n", + " params[\"starttime\"] = start\n", + " if end:\n", + " params[\"endtime\"] = end\n", + "\n", + " r = requests.get(f\"{BASE_URL}/gps/data/position/{station_code}/v3\", params=params)\n", + " r.raise_for_status()\n", + " \n", + " os.makedirs(SAVE_DIR, exist_ok=True)\n", + " filepath = os.path.join(SAVE_DIR, f\"{station_code}.csv\")\n", + " with open(filepath, \"w\") as f:\n", + " f.write(r.text)\n", + " # print(f\" Saved to {filepath}\")\n", + " return r.text\n", + "\n", + "\n", + "def parse_position_csv(csv_text):\n", + " lines = [l for l in csv_text.splitlines() if not l.startswith('#')]\n", + " df = pd.read_csv(StringIO(\"\\n\".join(lines)))\n", + " df.columns = df.columns.str.strip()\n", + " df[\"Datetime\"] = pd.to_datetime(df[\"Datetime\"])\n", + " return df" + ] + }, + { + "cell_type": "markdown", + "id": "65bc6ece-a770-49a1-9963-f3fcdd854ef1", + "metadata": {}, + "source": [ + "Before fetching all stations, we test on a single station to verify the response format and confirm our parameters are working correctly." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cea9a987-b648-487e-87bb-b44febbe9597", + "metadata": {}, + "outputs": [], + "source": [ + "# fetch and inspect a single station before running all stations\n", + "code = stations[\"ID\"].iloc[0]\n", + "print(f\"Fetching {code}...\")\n", + "pos_csv = get_position_timeseries(code, start=START, end=END)\n", + "df = parse_position_csv(pos_csv)\n", + "print(f\"{len(df)} epochs\")\n", + "df.head()" + ] + }, + { + "cell_type": "markdown", + "id": "f4568f27-e5a4-424e-991e-b6550f4efff3", + "metadata": {}, + "source": [ + "You should see a DataFrame with columns: `Datetime`, `delta N`, `delta E`, `delta U`, and associated standard deviations. Each row is one daily position estimate. " + ] + }, + { + "cell_type": "markdown", + "id": "e436d329-632a-4258-8253-65bc7bd7201b", + "metadata": {}, + "source": [ + "## Retrieving Data for All Stations" + ] + }, + { + "cell_type": "markdown", + "id": "50e7502c-fcff-46c1-a79f-c9aa72857ecc", + "metadata": {}, + "source": [ + "Now we fetch position time series for all stations. Stations that return a 404 error will be skipped automatically. See the Troubleshooting section for guidance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "38e6a91f-c756-44e8-94b2-f6c85ff97cc8", + "metadata": {}, + "outputs": [], + "source": [ + "for _, row in stations.iterrows():\n", + " code = row[\"ID\"]\n", + " print(f\"Fetching {code}...\")\n", + " try:\n", + " pos_csv = get_position_timeseries(code, start=START, end=END)\n", + " df = parse_position_csv(pos_csv)\n", + " print(f\" {len(df)} epochs\")\n", + " except requests.HTTPError as e:\n", + " print(f\" Skipping {code}: {e}\")" + ] + }, + { + "cell_type": "markdown", + "id": "10ba2f97-b3cd-4d6c-b322-c54c8e1372d0", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "Let's print a summary of what was retrieved to confirm everything looks as expected \n", + "before moving on to analysis." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ed8d0758-0a62-4145-9739-0f0cca2e244d", + "metadata": {}, + "outputs": [], + "source": [ + "successful = []\n", + "failed = []\n", + "\n", + "for _, row in stations.iterrows():\n", + " code = row[\"ID\"]\n", + " filepath = os.path.join(SAVE_DIR, f\"{code}.csv\")\n", + " if os.path.exists(filepath):\n", + " successful.append(code)\n", + " else:\n", + " failed.append(code)\n", + "\n", + "print(f\"Total stations found: {len(stations)}\")\n", + "print(f\"Successfully retrieved: {len(successful)}\")\n", + "print(f\"Failed: {len(failed)}\")\n", + "\n", + "if failed:\n", + " print(f\"\\nFailed stations: {failed}\")\n", + "\n", + "print(f\"\\nData saved to: {SAVE_DIR}\")" + ] + }, + { + "cell_type": "markdown", + "id": "3db5fdc7-54c3-4588-b685-e6aeda88cf6f", + "metadata": {}, + "source": [ + "## Troubleshooting\n", + "\n", + "- If a station returns a 404 error, for example `Skipping station_ID: 404 Client Error` it exists in the metadata but has no processed position solution for your date range. This is normal and the station will be skipped automatically. One solution might be to change the analysis center, for example from `cwu` to `unr` to see if another analysis center has processed the data." + ] + }, + { + "cell_type": "markdown", + "id": "811c5c9a-beb4-4293-8a47-67a809dfb39a", + "metadata": {}, + "source": [ + "## Try It Yourself\n", + "\n", + "- Change `MINLAT`, `MAXLAT`, `MINLON`, `MAXLON` to a different region and re-run\n", + "- Change `START`, `END` to a different time range and re-run\n", + "- Change `report` from `short` to `long`. What additional columns appear?" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 08380b8a33c9216011e849afcb2b1485f591202a Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Sun, 12 Jul 2026 21:46:32 +0000 Subject: [PATCH 2/6] Add notebook 1 from revised roadmap --- .../NB1-access-gnss-via-SDK-checkpoint.ipynb | 614 ++++++++++++++++++ .../NB1-access-gnss-via-SDK.ipynb | 614 ++++++++++++++++++ .../NB1-gnss-discovery_retrieval.ipynb | 602 ----------------- 3 files changed, 1228 insertions(+), 602 deletions(-) create mode 100644 tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb create mode 100644 tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb delete mode 100644 tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb diff --git a/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb new file mode 100644 index 0000000..e3b7dd5 --- /dev/null +++ b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb @@ -0,0 +1,614 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2abe5d1d-4a2f-4c96-a78c-6c3e5f9ac750", + "metadata": {}, + "source": [ + "# Accessing GNSS Observations with the EarthScope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "35320cb4-aed2-4420-8f9c-b3deb365e792", + "metadata": {}, + "source": [ + "**Version:** 1.0 | **Last updated:** 2026-07-09 | **Author:** Eshanta Mishra" + ] + }, + { + "cell_type": "markdown", + "id": "b9b1d1c3-569f-459a-8595-e1ba758e4bc7", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", + "\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to reterive GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "\n", + "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", + "* instantiating the EarthScope SDK client\n", + "* Select data for stations using the station name and time range.\n", + "* Retrieve observations and load into dataframes.\n", + "* Filter (slice) the GNSS data based on different parameters.\n", + "\n", + "---\n", + "\n", + "### Prerequisites\n", + "\n", + "\n", + "Before starting this notebook, you should:\n", + "* [ ] This is an introductory notebook. Being familiar with **Geodesy, GNSS, python dataframes** is recommended but not required.\n", + "\n", + "---\n", + "\n", + "### GeoLab Compute Resources\n", + "\n", + "| Setting | Recommended |\n", + "|---|---|\n", + "| **Image** | GeoLab (default image)|\n", + "| **Server size** | 4 GB RAM, ~0.5 CPUs (default server) |" + ] + }, + { + "cell_type": "markdown", + "id": "811fc68e-1e81-482c-82dd-5efa2a89d715", + "metadata": {}, + "source": [ + "## Learning Objectives\n", + "\n", + "By the end of this notebook, you will be able to:\n", + "\n", + "1. Instantiate EarthScope SDK client\n", + "2. Retreive and filter GNSS data from EarthScope" + ] + }, + { + "cell_type": "markdown", + "id": "1ec3d79a-ad70-4c58-b8fc-287501adb7e9", + "metadata": {}, + "source": [ + "## Relevant Documentation & Resources\n", + "\n", + "* [EarthScope SDK documentation](https://docs.earthscope.org/sdk)" + ] + }, + { + "cell_type": "markdown", + "id": "076d400d-8c88-4113-a28d-91dac8648679", + "metadata": {}, + "source": [ + "## Contents\n", + "\n", + "1. [Basics of GNSS](#id-1-basics-of-gnss)\n", + "2. [Setup & Imports](#id-2-setup-imports)\n", + "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", + "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", + "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", + "6. [Larger than Memory Requests with Query Plan](#id-6-larger-than-memory-requests-with-query-plans)\n", + "7. [Exploration Exercises](#id-7-exploration-exercises)\n", + "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" + ] + }, + { + "cell_type": "markdown", + "id": "1b245a5d-5792-461d-bd51-99e55ab797a0", + "metadata": {}, + "source": [ + "## 1. Basics of GNSS" + ] + }, + { + "cell_type": "markdown", + "id": "9e03f7ee-00b4-4501-bd77-551d49ca52a6", + "metadata": {}, + "source": [ + "A GNSS (Global Navigation Satellite System) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", + "\n", + "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", + "\n", + "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", + "2. **Carrier phase**: It is a more precise distance measurement between the satellite and thr ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", + "3. **SNR**: Signal-to-Noise Ratio. IT is a parameter that tells how strong and clean the received signal is.\n", + "\n", + "**Raw Observations vs. derived products**\n", + "\n", + "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " + ] + }, + { + "cell_type": "markdown", + "id": "cca79b91-0348-4ee2-842e-740595288820", + "metadata": {}, + "source": [ + "## 2. Setup & Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8cd2903c-1b18-4877-a421-f19374f8fcf3", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime as dt\n", + "import polars as pl\n", + "from earthscope_sdk import AsyncEarthScopeClient" + ] + }, + { + "cell_type": "markdown", + "id": "50263265-10c7-44c7-a302-37b658b899c8", + "metadata": {}, + "source": [ + "### Instantiating the Earthscope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "8683b5f9-3ffc-4eb4-a8ec-82c61fa196e7", + "metadata": {}, + "source": [ + "#### Authentication" + ] + }, + { + "cell_type": "markdown", + "id": "52c5e185-8dca-42ff-8a9d-89c23297a761", + "metadata": {}, + "source": [ + "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are always available inside it. So, the client authenticates automatically. This removes the step of logging in or passing tokens manually, as you would do in a non GeoLab Environment." + ] + }, + { + "cell_type": "markdown", + "id": "b0143b85-f435-4ce2-9359-6b5f2b8d3674", + "metadata": {}, + "source": [ + "#### Async client" + ] + }, + { + "cell_type": "markdown", + "id": "421226b8-42bc-4d91-a517-469c29fa6e85", + "metadata": {}, + "source": [ + "`AsyncEarthScopeClient` is the asynchronous client. Async lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "abe833e0-9ca9-48bd-8bfc-749579592f92", + "metadata": {}, + "outputs": [], + "source": [ + "es = AsyncEarthScopeClient()" + ] + }, + { + "cell_type": "markdown", + "id": "aa3e8c77-ceaf-4f6e-be2e-b99b0b7f0125", + "metadata": {}, + "source": [ + "### Configuration\n", + "\n", + "Set your parameters here before running the rest of the notebook. Every subsequent cells read from these variables. So, this is the only place you need to edit to point the notebook at different data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15ecb388-d718-4949-a072-5db6cd398d65", + "metadata": {}, + "outputs": [], + "source": [ + "# Modify these values before running the notebook.\n", + "\n", + "STATION = \"AC60\" # GNSS station (4-character ID)\n", + "SESSION = \"A\" # Session name\n", + "START = dt.datetime(2025, 7, 20, 21) # Query start (UTC)\n", + "END = dt.datetime(2025, 7, 21, 3) # Query end (UTC)" + ] + }, + { + "cell_type": "markdown", + "id": "f2e9e575-a7b1-4dd8-86a9-17f0fd942f15", + "metadata": {}, + "source": [ + "## 3. Retrieve Observations for a single station" + ] + }, + { + "cell_type": "markdown", + "id": "af77148f-64a0-424a-9dc2-2ec34a8f96e8", + "metadata": {}, + "source": [ + "In this step, we will retrieve GNSS observations for a single station and time window as an Apache arrow table through the EarthScope data API.\n", + "\n", + "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", + "\n", + "The Expected output is a single table with one row per satellite, per signal, per epoch" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4eb07e18-b72b-4ed4-99b7-558d935fee31", + "metadata": {}, + "outputs": [], + "source": [ + "# Step: describe the request, then .fetch() to run the query.\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=START,\n", + " end_datetime=END,\n", + " station_name=STATION,\n", + " session_name=SESSION,\n", + ").fetch()\n", + "\n", + "table" + ] + }, + { + "cell_type": "markdown", + "id": "b5d3fab2-0ab0-4231-9681-b6e56a117f9d", + "metadata": {}, + "source": [ + "Convert the Arrow table to a Polars dataframe with `pl.from_arrow(...)`. This is zero-copy, so it is very efficient. Sorting by `timestamp` makes the rows read chronologically." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "82b54655-559f-4b95-8479-a386affe5b91", + "metadata": {}, + "outputs": [], + "source": [ + "df = pl.from_arrow(table).sort(\"timestamp\")\n", + "df" + ] + }, + { + "cell_type": "markdown", + "id": "05678a5b-388b-4d44-8e04-28a8912af92e", + "metadata": {}, + "source": [ + "## 4. Understanding the Observation fields" + ] + }, + { + "cell_type": "markdown", + "id": "6f775aef-3b48-4a78-9d4f-b43b7cd1c143", + "metadata": {}, + "source": [ + "The default data query returns every field - the same information you would find in a RINEX observation file. Let us take a moment to understand the data structure.\n", + "Each row of data is a single measurement (one satellite, one signal, one instant). Inspect the columns, the constellations present, and the range of signal strengths." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1ab79788-7803-4711-aa4f-f977b8e83e7a", + "metadata": {}, + "outputs": [], + "source": [ + "# Inspect structure and coverage\n", + "print(df.schema) # column names and types\n", + "print(\"Constellations:\", df[\"system\"].unique().to_list())\n", + "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())\n", + "\n", + "df.describe() # summary statistics" + ] + }, + { + "cell_type": "markdown", + "id": "c59fb35b-ec60-4879-8a0e-1c6bbe9e4a4a", + "metadata": {}, + "source": [ + "### What each field means\n", + "\n", + "| Column | Type | Meaning |\n", + "|---|---|---|\n", + "| `timestamp` | datetime (UTC) | The epoch of the observation. |\n", + "| `satellite` | int | Satellite number within its constellation (its PRN / slot). Combine with `system` for a globally unique ID. |\n", + "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L`, `5Q` (decoded below). |\n", + "| `range` | float | Pseudorange in meters. It is the apparent satellite to receiver distance. |\n", + "| `phase` | float | Carrier phase in cycles. It is a precise but ambiguous range measurement. |\n", + "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher is cleaner. |\n", + "| `slip` | int | Cycle-slip indicator; `null` when phase tracking was continuous. |\n", + "| `flags` | int | Per-observation status flags (e.g. loss-of-lock). |\n", + "| `fcn` | int | Frequency channel number — used by GLONASS's FDMA signals; `0` otherwise. |\n", + "| `system` | str | Constellation code (see below). |\n", + "| `igs` | str | The station's IGS long name (e.g. `AC6000USA`). |\n", + "\n", + "### Reading the `system` column\n", + "\n", + "| Code | Constellation |\n", + "|---|---|\n", + "| `G` | GPS (USA) |\n", + "| `R` | GLONASS (Russia) |\n", + "| `E` | Galileo (EU) |\n", + "| `C` | BeiDou (China) |\n", + "| `J` | QZSS (Japan) |\n", + "| `I` | NavIC / IRNSS (India) |\n", + "| `S` | SBAS (augmentation) |\n", + "\n", + "### Reading the `obs_code` column\n", + "\n", + "An observation code names a specific signal as **band + tracking attribute**:\n", + "\n", + "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", + "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", + "\n", + "Fpr example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." + ] + }, + { + "cell_type": "markdown", + "id": "7f1cd227-e581-4790-97a7-176bfc846f2a", + "metadata": {}, + "source": [ + "## 5. Requesting Only the Data You Need" + ] + }, + { + "cell_type": "markdown", + "id": "c0a4e066-7b6b-4161-9260-a4a825637f2a", + "metadata": {}, + "source": [ + "When you are working with RINEX files, the files hand you everything, all fields, constellations, satellites etc. Most analyses only need a small slice of these data. Passing filters to `gnss_observations()` in the SDK pushes that selection to the server, so only the data that you asked for is transferred.\n", + "\n", + "Downloading a single signal from a single satellite over two months is drastically cheaper than downloading full RINEX and discarding 99% of it. Less data means faster fetches and smaller memory footprints.\n", + "\n", + "The resulting output is a dataframe containing only the requested stations, constellations, satellites, signals and fields. For example, only `snr` and `range` columns when `field=[\"snr\",\"range\"]`.\n", + "\n", + "Each filter below is optional, accepts a single value or a list, and shrinks the request:\n", + "\n", + "* `station_name`: one name or a list of names\n", + "* `network_name`: request a whole network at once\n", + "* `system`: constellation code(s), e.g. `\"G\"` or `[\"G\", \"R\"]`\n", + "* `satellite`: specific satellite number(s)\n", + "* `obs_code`: specific signal(s), e.g. `[\"1C\", \"2L\"]`\n", + "* `field`: which measurement column(s) to return, e.g. `[\"snr\", \"range\"]`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "949a6db5-f949-43f9-930c-88344765e60b", + "metadata": {}, + "outputs": [], + "source": [ + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " station_name=[\"P717\", \"P453\", \"P146\", \"P147\", \"P041\"],\n", + " session_name=\"A\",\n", + " system=[\"G\", \"R\"],\n", + " obs_code=[\"1C\", \"2L\"],\n", + " satellite=[\"7\", \"21\", \"28\"],\n", + " field=[\"snr\", \"range\"],\n", + ").fetch()\n", + "\n", + "df_sliced = pl.from_arrow(table).sort(\"timestamp\")\n", + "df_sliced" + ] + }, + { + "cell_type": "markdown", + "id": "0cad076e-a951-416e-ad88-96dd7fe1f253", + "metadata": {}, + "source": [ + "> **Check:** The result should contain only the `snr` and `range` measurement columns, no `phase`, no `flags`." + ] + }, + { + "cell_type": "markdown", + "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", + "metadata": {}, + "source": [ + "## 6. Larger than Memory Requests with Query Plan" + ] + }, + { + "cell_type": "markdown", + "id": "81e989fa-baf0-4396-920e-fece9f620f43", + "metadata": {}, + "source": [ + "### What is a query plan?" + ] + }, + { + "cell_type": "markdown", + "id": "7a61354a-548f-4459-976c-89d94b1d5104", + "metadata": {}, + "source": [ + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageble groups such as by day or by station. This allows you to proccess one group at a time.\n", + "\n", + "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", + "\n", + "The expected output is a summary for each group (row count, time span, and stations), rather than one huge dataframe." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1772dd83-792b-4666-a107-392d8522b92f", + "metadata": {}, + "outputs": [], + "source": [ + "# Build a query plan (note: no .fetch())\n", + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "plan = es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " network_name=\"PERM:Alaska\",\n", + " session_name=\"A\",\n", + " system=\"G\",\n", + " field=[\"phase\", \"range\", \"snr\"],\n", + ")\n", + "\n", + "print(plan) # preview the plan's request/group counts before any data is fetched\n", + "\n", + "\n", + "def summarize(table):\n", + " # Print a quick summary of one group's table.\n", + " d = pl.from_arrow(table)\n", + " stations = d[\"igs\"].unique().sort().to_list()\n", + " if len(stations) > 10:\n", + " stations = f\"{len(stations)} stations\"\n", + " print(len(d), d[\"timestamp\"].min(), d[\"timestamp\"].max(), stations)" + ] + }, + { + "cell_type": "markdown", + "id": "6f59fcbd-b77a-4733-82cc-b064b081bc66", + "metadata": {}, + "source": [ + "Printing a plan reports how many API requests and groups it will run before any data moves. This provides the users with a quick way to gauge how large a query is." + ] + }, + { + "cell_type": "markdown", + "id": "ac24a294-a48d-416a-a285-a2abd1f633c5", + "metadata": {}, + "source": [ + "Processing the plan one day at a time. You never hold more than a single day in memory at once." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "46d1af54-622f-4066-9895-cfd595341cdd", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_day():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "02c76245-1e7a-4663-b398-f0472a58eb68", + "metadata": {}, + "source": [ + "Processing the plan one station at a time across the whole window instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "34742255-bd16-4197-b151-1777b4753a1d", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_station():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", + "metadata": {}, + "source": [ + " > check: Each line of output represents one group." + ] + }, + { + "cell_type": "markdown", + "id": "8e88b724-65ec-4ef7-9e19-fb6bbf8c895b", + "metadata": {}, + "source": [ + "### A rough guide for choosing a strategy" + ] + }, + { + "cell_type": "markdown", + "id": "d0673a80-e5e9-4256-8f3f-a48ba2ddafa2", + "metadata": {}, + "source": [ + "* Less than a week: `.fetch()` the whole results at once.\n", + "* Weeks to months, or many stations: iterate with `group_by_day()` (or `group_by_station()`)\n", + "* Months to years: define custom batches with `plan.group_by()`." + ] + }, + { + "cell_type": "markdown", + "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", + "metadata": {}, + "source": [ + "For cutom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [Earthscope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." + ] + }, + { + "cell_type": "markdown", + "id": "c5008800-7811-4fbc-9721-9869da927fc2", + "metadata": {}, + "source": [ + "## 7. Exploration Exercises\n", + "\n", + "Now that you've completed the core workflow, try modifying the parameters below to explore how the results change.\n", + "\n", + "**Try these modifications:**\n", + "\n", + "1. **Change the station:** Set `STATION` in the Configuration section to a different 4-character station ID and re-run Section 3. Does the station report the same constellations?\n", + "\n", + "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"5Q\"`) with `field=\"snr\"`. How much smaller is the result?\n", + "\n", + "3. **Save your results:** Write a fetched dataframe to the scratch directory (*Hint: Use `os.environ[\"SCRATCH_BUCKET\"]`*) as Parquet with `df.write_parquet(...)`, then read it back. Parquet preserves types and is far smaller than CSV." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "edf9b94a-b045-4fcf-be34-f2e2f958f124", + "metadata": {}, + "outputs": [], + "source": [ + "# Exploration cell — use this space to experiment" + ] + }, + { + "cell_type": "markdown", + "id": "1e147f5a-2352-4a16-b28b-3f5c1dbea902", + "metadata": {}, + "source": [ + "## 8. Troubleshooting & Support\n", + "\n", + "### Further Resources\n", + "\n", + "* [EarthScope SDK Documentation](https://docs.earthscope.org/sdk)\n", + "* [SDK GNSS Observation tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", + "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", + "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb new file mode 100644 index 0000000..e3b7dd5 --- /dev/null +++ b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb @@ -0,0 +1,614 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2abe5d1d-4a2f-4c96-a78c-6c3e5f9ac750", + "metadata": {}, + "source": [ + "# Accessing GNSS Observations with the EarthScope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "35320cb4-aed2-4420-8f9c-b3deb365e792", + "metadata": {}, + "source": [ + "**Version:** 1.0 | **Last updated:** 2026-07-09 | **Author:** Eshanta Mishra" + ] + }, + { + "cell_type": "markdown", + "id": "b9b1d1c3-569f-459a-8595-e1ba758e4bc7", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", + "\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to reterive GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "\n", + "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", + "* instantiating the EarthScope SDK client\n", + "* Select data for stations using the station name and time range.\n", + "* Retrieve observations and load into dataframes.\n", + "* Filter (slice) the GNSS data based on different parameters.\n", + "\n", + "---\n", + "\n", + "### Prerequisites\n", + "\n", + "\n", + "Before starting this notebook, you should:\n", + "* [ ] This is an introductory notebook. Being familiar with **Geodesy, GNSS, python dataframes** is recommended but not required.\n", + "\n", + "---\n", + "\n", + "### GeoLab Compute Resources\n", + "\n", + "| Setting | Recommended |\n", + "|---|---|\n", + "| **Image** | GeoLab (default image)|\n", + "| **Server size** | 4 GB RAM, ~0.5 CPUs (default server) |" + ] + }, + { + "cell_type": "markdown", + "id": "811fc68e-1e81-482c-82dd-5efa2a89d715", + "metadata": {}, + "source": [ + "## Learning Objectives\n", + "\n", + "By the end of this notebook, you will be able to:\n", + "\n", + "1. Instantiate EarthScope SDK client\n", + "2. Retreive and filter GNSS data from EarthScope" + ] + }, + { + "cell_type": "markdown", + "id": "1ec3d79a-ad70-4c58-b8fc-287501adb7e9", + "metadata": {}, + "source": [ + "## Relevant Documentation & Resources\n", + "\n", + "* [EarthScope SDK documentation](https://docs.earthscope.org/sdk)" + ] + }, + { + "cell_type": "markdown", + "id": "076d400d-8c88-4113-a28d-91dac8648679", + "metadata": {}, + "source": [ + "## Contents\n", + "\n", + "1. [Basics of GNSS](#id-1-basics-of-gnss)\n", + "2. [Setup & Imports](#id-2-setup-imports)\n", + "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", + "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", + "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", + "6. [Larger than Memory Requests with Query Plan](#id-6-larger-than-memory-requests-with-query-plans)\n", + "7. [Exploration Exercises](#id-7-exploration-exercises)\n", + "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" + ] + }, + { + "cell_type": "markdown", + "id": "1b245a5d-5792-461d-bd51-99e55ab797a0", + "metadata": {}, + "source": [ + "## 1. Basics of GNSS" + ] + }, + { + "cell_type": "markdown", + "id": "9e03f7ee-00b4-4501-bd77-551d49ca52a6", + "metadata": {}, + "source": [ + "A GNSS (Global Navigation Satellite System) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", + "\n", + "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", + "\n", + "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", + "2. **Carrier phase**: It is a more precise distance measurement between the satellite and thr ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", + "3. **SNR**: Signal-to-Noise Ratio. IT is a parameter that tells how strong and clean the received signal is.\n", + "\n", + "**Raw Observations vs. derived products**\n", + "\n", + "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " + ] + }, + { + "cell_type": "markdown", + "id": "cca79b91-0348-4ee2-842e-740595288820", + "metadata": {}, + "source": [ + "## 2. Setup & Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8cd2903c-1b18-4877-a421-f19374f8fcf3", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime as dt\n", + "import polars as pl\n", + "from earthscope_sdk import AsyncEarthScopeClient" + ] + }, + { + "cell_type": "markdown", + "id": "50263265-10c7-44c7-a302-37b658b899c8", + "metadata": {}, + "source": [ + "### Instantiating the Earthscope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "8683b5f9-3ffc-4eb4-a8ec-82c61fa196e7", + "metadata": {}, + "source": [ + "#### Authentication" + ] + }, + { + "cell_type": "markdown", + "id": "52c5e185-8dca-42ff-8a9d-89c23297a761", + "metadata": {}, + "source": [ + "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are always available inside it. So, the client authenticates automatically. This removes the step of logging in or passing tokens manually, as you would do in a non GeoLab Environment." + ] + }, + { + "cell_type": "markdown", + "id": "b0143b85-f435-4ce2-9359-6b5f2b8d3674", + "metadata": {}, + "source": [ + "#### Async client" + ] + }, + { + "cell_type": "markdown", + "id": "421226b8-42bc-4d91-a517-469c29fa6e85", + "metadata": {}, + "source": [ + "`AsyncEarthScopeClient` is the asynchronous client. Async lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "abe833e0-9ca9-48bd-8bfc-749579592f92", + "metadata": {}, + "outputs": [], + "source": [ + "es = AsyncEarthScopeClient()" + ] + }, + { + "cell_type": "markdown", + "id": "aa3e8c77-ceaf-4f6e-be2e-b99b0b7f0125", + "metadata": {}, + "source": [ + "### Configuration\n", + "\n", + "Set your parameters here before running the rest of the notebook. Every subsequent cells read from these variables. So, this is the only place you need to edit to point the notebook at different data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15ecb388-d718-4949-a072-5db6cd398d65", + "metadata": {}, + "outputs": [], + "source": [ + "# Modify these values before running the notebook.\n", + "\n", + "STATION = \"AC60\" # GNSS station (4-character ID)\n", + "SESSION = \"A\" # Session name\n", + "START = dt.datetime(2025, 7, 20, 21) # Query start (UTC)\n", + "END = dt.datetime(2025, 7, 21, 3) # Query end (UTC)" + ] + }, + { + "cell_type": "markdown", + "id": "f2e9e575-a7b1-4dd8-86a9-17f0fd942f15", + "metadata": {}, + "source": [ + "## 3. Retrieve Observations for a single station" + ] + }, + { + "cell_type": "markdown", + "id": "af77148f-64a0-424a-9dc2-2ec34a8f96e8", + "metadata": {}, + "source": [ + "In this step, we will retrieve GNSS observations for a single station and time window as an Apache arrow table through the EarthScope data API.\n", + "\n", + "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", + "\n", + "The Expected output is a single table with one row per satellite, per signal, per epoch" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4eb07e18-b72b-4ed4-99b7-558d935fee31", + "metadata": {}, + "outputs": [], + "source": [ + "# Step: describe the request, then .fetch() to run the query.\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=START,\n", + " end_datetime=END,\n", + " station_name=STATION,\n", + " session_name=SESSION,\n", + ").fetch()\n", + "\n", + "table" + ] + }, + { + "cell_type": "markdown", + "id": "b5d3fab2-0ab0-4231-9681-b6e56a117f9d", + "metadata": {}, + "source": [ + "Convert the Arrow table to a Polars dataframe with `pl.from_arrow(...)`. This is zero-copy, so it is very efficient. Sorting by `timestamp` makes the rows read chronologically." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "82b54655-559f-4b95-8479-a386affe5b91", + "metadata": {}, + "outputs": [], + "source": [ + "df = pl.from_arrow(table).sort(\"timestamp\")\n", + "df" + ] + }, + { + "cell_type": "markdown", + "id": "05678a5b-388b-4d44-8e04-28a8912af92e", + "metadata": {}, + "source": [ + "## 4. Understanding the Observation fields" + ] + }, + { + "cell_type": "markdown", + "id": "6f775aef-3b48-4a78-9d4f-b43b7cd1c143", + "metadata": {}, + "source": [ + "The default data query returns every field - the same information you would find in a RINEX observation file. Let us take a moment to understand the data structure.\n", + "Each row of data is a single measurement (one satellite, one signal, one instant). Inspect the columns, the constellations present, and the range of signal strengths." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1ab79788-7803-4711-aa4f-f977b8e83e7a", + "metadata": {}, + "outputs": [], + "source": [ + "# Inspect structure and coverage\n", + "print(df.schema) # column names and types\n", + "print(\"Constellations:\", df[\"system\"].unique().to_list())\n", + "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())\n", + "\n", + "df.describe() # summary statistics" + ] + }, + { + "cell_type": "markdown", + "id": "c59fb35b-ec60-4879-8a0e-1c6bbe9e4a4a", + "metadata": {}, + "source": [ + "### What each field means\n", + "\n", + "| Column | Type | Meaning |\n", + "|---|---|---|\n", + "| `timestamp` | datetime (UTC) | The epoch of the observation. |\n", + "| `satellite` | int | Satellite number within its constellation (its PRN / slot). Combine with `system` for a globally unique ID. |\n", + "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L`, `5Q` (decoded below). |\n", + "| `range` | float | Pseudorange in meters. It is the apparent satellite to receiver distance. |\n", + "| `phase` | float | Carrier phase in cycles. It is a precise but ambiguous range measurement. |\n", + "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher is cleaner. |\n", + "| `slip` | int | Cycle-slip indicator; `null` when phase tracking was continuous. |\n", + "| `flags` | int | Per-observation status flags (e.g. loss-of-lock). |\n", + "| `fcn` | int | Frequency channel number — used by GLONASS's FDMA signals; `0` otherwise. |\n", + "| `system` | str | Constellation code (see below). |\n", + "| `igs` | str | The station's IGS long name (e.g. `AC6000USA`). |\n", + "\n", + "### Reading the `system` column\n", + "\n", + "| Code | Constellation |\n", + "|---|---|\n", + "| `G` | GPS (USA) |\n", + "| `R` | GLONASS (Russia) |\n", + "| `E` | Galileo (EU) |\n", + "| `C` | BeiDou (China) |\n", + "| `J` | QZSS (Japan) |\n", + "| `I` | NavIC / IRNSS (India) |\n", + "| `S` | SBAS (augmentation) |\n", + "\n", + "### Reading the `obs_code` column\n", + "\n", + "An observation code names a specific signal as **band + tracking attribute**:\n", + "\n", + "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", + "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", + "\n", + "Fpr example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." + ] + }, + { + "cell_type": "markdown", + "id": "7f1cd227-e581-4790-97a7-176bfc846f2a", + "metadata": {}, + "source": [ + "## 5. Requesting Only the Data You Need" + ] + }, + { + "cell_type": "markdown", + "id": "c0a4e066-7b6b-4161-9260-a4a825637f2a", + "metadata": {}, + "source": [ + "When you are working with RINEX files, the files hand you everything, all fields, constellations, satellites etc. Most analyses only need a small slice of these data. Passing filters to `gnss_observations()` in the SDK pushes that selection to the server, so only the data that you asked for is transferred.\n", + "\n", + "Downloading a single signal from a single satellite over two months is drastically cheaper than downloading full RINEX and discarding 99% of it. Less data means faster fetches and smaller memory footprints.\n", + "\n", + "The resulting output is a dataframe containing only the requested stations, constellations, satellites, signals and fields. For example, only `snr` and `range` columns when `field=[\"snr\",\"range\"]`.\n", + "\n", + "Each filter below is optional, accepts a single value or a list, and shrinks the request:\n", + "\n", + "* `station_name`: one name or a list of names\n", + "* `network_name`: request a whole network at once\n", + "* `system`: constellation code(s), e.g. `\"G\"` or `[\"G\", \"R\"]`\n", + "* `satellite`: specific satellite number(s)\n", + "* `obs_code`: specific signal(s), e.g. `[\"1C\", \"2L\"]`\n", + "* `field`: which measurement column(s) to return, e.g. `[\"snr\", \"range\"]`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "949a6db5-f949-43f9-930c-88344765e60b", + "metadata": {}, + "outputs": [], + "source": [ + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " station_name=[\"P717\", \"P453\", \"P146\", \"P147\", \"P041\"],\n", + " session_name=\"A\",\n", + " system=[\"G\", \"R\"],\n", + " obs_code=[\"1C\", \"2L\"],\n", + " satellite=[\"7\", \"21\", \"28\"],\n", + " field=[\"snr\", \"range\"],\n", + ").fetch()\n", + "\n", + "df_sliced = pl.from_arrow(table).sort(\"timestamp\")\n", + "df_sliced" + ] + }, + { + "cell_type": "markdown", + "id": "0cad076e-a951-416e-ad88-96dd7fe1f253", + "metadata": {}, + "source": [ + "> **Check:** The result should contain only the `snr` and `range` measurement columns, no `phase`, no `flags`." + ] + }, + { + "cell_type": "markdown", + "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", + "metadata": {}, + "source": [ + "## 6. Larger than Memory Requests with Query Plan" + ] + }, + { + "cell_type": "markdown", + "id": "81e989fa-baf0-4396-920e-fece9f620f43", + "metadata": {}, + "source": [ + "### What is a query plan?" + ] + }, + { + "cell_type": "markdown", + "id": "7a61354a-548f-4459-976c-89d94b1d5104", + "metadata": {}, + "source": [ + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageble groups such as by day or by station. This allows you to proccess one group at a time.\n", + "\n", + "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", + "\n", + "The expected output is a summary for each group (row count, time span, and stations), rather than one huge dataframe." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1772dd83-792b-4666-a107-392d8522b92f", + "metadata": {}, + "outputs": [], + "source": [ + "# Build a query plan (note: no .fetch())\n", + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "plan = es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " network_name=\"PERM:Alaska\",\n", + " session_name=\"A\",\n", + " system=\"G\",\n", + " field=[\"phase\", \"range\", \"snr\"],\n", + ")\n", + "\n", + "print(plan) # preview the plan's request/group counts before any data is fetched\n", + "\n", + "\n", + "def summarize(table):\n", + " # Print a quick summary of one group's table.\n", + " d = pl.from_arrow(table)\n", + " stations = d[\"igs\"].unique().sort().to_list()\n", + " if len(stations) > 10:\n", + " stations = f\"{len(stations)} stations\"\n", + " print(len(d), d[\"timestamp\"].min(), d[\"timestamp\"].max(), stations)" + ] + }, + { + "cell_type": "markdown", + "id": "6f59fcbd-b77a-4733-82cc-b064b081bc66", + "metadata": {}, + "source": [ + "Printing a plan reports how many API requests and groups it will run before any data moves. This provides the users with a quick way to gauge how large a query is." + ] + }, + { + "cell_type": "markdown", + "id": "ac24a294-a48d-416a-a285-a2abd1f633c5", + "metadata": {}, + "source": [ + "Processing the plan one day at a time. You never hold more than a single day in memory at once." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "46d1af54-622f-4066-9895-cfd595341cdd", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_day():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "02c76245-1e7a-4663-b398-f0472a58eb68", + "metadata": {}, + "source": [ + "Processing the plan one station at a time across the whole window instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "34742255-bd16-4197-b151-1777b4753a1d", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_station():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", + "metadata": {}, + "source": [ + " > check: Each line of output represents one group." + ] + }, + { + "cell_type": "markdown", + "id": "8e88b724-65ec-4ef7-9e19-fb6bbf8c895b", + "metadata": {}, + "source": [ + "### A rough guide for choosing a strategy" + ] + }, + { + "cell_type": "markdown", + "id": "d0673a80-e5e9-4256-8f3f-a48ba2ddafa2", + "metadata": {}, + "source": [ + "* Less than a week: `.fetch()` the whole results at once.\n", + "* Weeks to months, or many stations: iterate with `group_by_day()` (or `group_by_station()`)\n", + "* Months to years: define custom batches with `plan.group_by()`." + ] + }, + { + "cell_type": "markdown", + "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", + "metadata": {}, + "source": [ + "For cutom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [Earthscope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." + ] + }, + { + "cell_type": "markdown", + "id": "c5008800-7811-4fbc-9721-9869da927fc2", + "metadata": {}, + "source": [ + "## 7. Exploration Exercises\n", + "\n", + "Now that you've completed the core workflow, try modifying the parameters below to explore how the results change.\n", + "\n", + "**Try these modifications:**\n", + "\n", + "1. **Change the station:** Set `STATION` in the Configuration section to a different 4-character station ID and re-run Section 3. Does the station report the same constellations?\n", + "\n", + "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"5Q\"`) with `field=\"snr\"`. How much smaller is the result?\n", + "\n", + "3. **Save your results:** Write a fetched dataframe to the scratch directory (*Hint: Use `os.environ[\"SCRATCH_BUCKET\"]`*) as Parquet with `df.write_parquet(...)`, then read it back. Parquet preserves types and is far smaller than CSV." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "edf9b94a-b045-4fcf-be34-f2e2f958f124", + "metadata": {}, + "outputs": [], + "source": [ + "# Exploration cell — use this space to experiment" + ] + }, + { + "cell_type": "markdown", + "id": "1e147f5a-2352-4a16-b28b-3f5c1dbea902", + "metadata": {}, + "source": [ + "## 8. Troubleshooting & Support\n", + "\n", + "### Further Resources\n", + "\n", + "* [EarthScope SDK Documentation](https://docs.earthscope.org/sdk)\n", + "* [SDK GNSS Observation tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", + "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", + "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb b/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb deleted file mode 100644 index 41e4e27..0000000 --- a/tutorials/mvp-1-geodesy/NB1-gnss-discovery_retrieval.ipynb +++ /dev/null @@ -1,602 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "3ef3d1da-4ad2-43d2-835b-cb36a8002baf", - "metadata": {}, - "source": [ - "# GNSS Station Discovery and Retrieval" - ] - }, - { - "cell_type": "markdown", - "id": "00a4248a-322c-42e7-96d2-231740c1af8a", - "metadata": {}, - "source": [ - "**Prerequisites:** Working knowledge of Python and Jupyter notebooks, understanding of GNSS data\n", - "\n", - "**GeoLab compute:** Default Image (4 GB RAM, ~0.5 CPU)" - ] - }, - { - "cell_type": "markdown", - "id": "5ce4783d-27e6-4c2d-93ee-b391b0de5c80", - "metadata": {}, - "source": [ - "## Overview\n", - "\n", - "This notebook demonstrates how to retrieve processed GNSS position time series data from the GAGE web services API provided by EarthScope. You will define a geographic study area, identify available stations, visualize them on an interactive map, and download position time series data to your scratch storage. The output can be used to quickly visualize the GNSS stations available in your Area of Interest (AOI) and download the position time series which can then be used for future workflows.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "id": "f2cdf0e7-2742-41da-a3c6-708b76ba6a07", - "metadata": { - "jp-MarkdownHeadingCollapsed": true - }, - "source": [ - "## Learning Objectives\n", - "\n", - "By the end of this notebook you will be able to:\n", - "\n", - "1. Query the GAGE API to find GNSS stations within a bounding box\n", - "2. Visualize station locations on an interactive map\n", - "3. Retrieve and save processed position time series data for multiple stations\n", - "4. Understand the structure of the GAGE GeoCSV response format" - ] - }, - { - "cell_type": "markdown", - "id": "dc127961-504d-42f6-b5a0-bedbd35665dc", - "metadata": {}, - "source": [ - "## Related Documentation\n", - "\n", - "- [GAGE Web Services Documentation](https://www.unavco.org/data/web-services/documentation/documentation.html#/GNSS47GPS)\n", - "- [GNSS Position Data Documentation](https://www.unavco.org/data/web-services/documentation/gps-position-documentation.html)" - ] - }, - { - "cell_type": "markdown", - "id": "98abba98-c457-45f5-a0af-dbe7ef0ed97e", - "metadata": {}, - "source": [ - "## Setup\n", - "We begin by importing the necessary Python libraries and setting the base URL for all API requests." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "28bde389-1e27-4b6d-a444-0a9078665461", - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import shutil\n", - "import requests\n", - "import pandas as pd\n", - "import folium\n", - "import matplotlib.pyplot as plt\n", - "from io import StringIO" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1f9efc80-809a-4157-b848-609648c002e5", - "metadata": {}, - "outputs": [], - "source": [ - "#base URL for GAGE/UNAVCO web service requests.\n", - "BASE_URL = \"https://web-services.unavco.org\"" - ] - }, - { - "cell_type": "markdown", - "id": "00368430-718e-4889-855e-35def3da3af5", - "metadata": {}, - "source": [ - "## Defining Study Area" - ] - }, - { - "cell_type": "markdown", - "id": "776ae7f7-1a99-4918-9b65-c5b66f5cea68", - "metadata": {}, - "source": [ - "We define our study area using a rectangular bounding box that is defined by minimum and maximum latitude and longitude (in degrees). We also define the time range we are interested in using the parameters `START` and `END` in the format `YYYY-MM-DD`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "936d3fc4-95a5-4baa-848c-59e77cb3e129", - "metadata": {}, - "outputs": [], - "source": [ - "MINLAT, MAXLAT = 43, 46\n", - "MINLON, MAXLON = -125, -123\n", - "START, END = \"2017-01-01\", \"2024-01-01\"" - ] - }, - { - "cell_type": "markdown", - "id": "c5c7a794-dd01-4784-a678-caabe9617de1", - "metadata": {}, - "source": [ - "Before retrieving data, we visualize the bounding box on an interactive map to make sure it covers the area we intend. This is a good sanity check to see our coordinates match our intended AOI before making API calls." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "eae9ef94-3716-4a99-9b90-d6df34dc5c8d", - "metadata": {}, - "outputs": [], - "source": [ - "def plot_bbox(minlat, maxlat, minlon, maxlon, height=400):\n", - " center_lat = (minlat + maxlat) / 2\n", - " center_lon = (minlon + maxlon) / 2\n", - "\n", - " fig = folium.Figure(height=height)\n", - " m = folium.Map(location=[center_lat, center_lon], zoom_start=7,\n", - " tiles=\"CartoDB positron\")\n", - " m.add_to(fig)\n", - "\n", - " folium.Rectangle(\n", - " bounds=[[minlat, minlon], [maxlat, maxlon]],\n", - " color=\"blue\",\n", - " fill=True,\n", - " fill_opacity=0.1,\n", - " tooltip=f\"Bounding box: ({minlat}, {minlon}) to ({maxlat}, {maxlon})\"\n", - " ).add_to(m)\n", - "\n", - " return fig" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "902eb26a-a826-4fb8-be8a-bc6d2016e2cc", - "metadata": {}, - "outputs": [], - "source": [ - "plot_bbox(MINLAT, MAXLAT, MINLON, MAXLON)" - ] - }, - { - "cell_type": "markdown", - "id": "fcbb166b-7528-4bd9-8a31-1ffb47421ec3", - "metadata": {}, - "source": [ - "## Retrieving Station Metadata" - ] - }, - { - "cell_type": "markdown", - "id": "f01da461-e200-4906-8ff7-3ae5586064b3", - "metadata": {}, - "source": [ - "The GAGE API endpoint `gps/metadata/sites/v1` retrieves site metadata for all GNSS sites that fall within a spatial bounding box defined by north and south latitude, and east and west longitude. The response is a GeoCSV string (a comma separated text format that contains geographic data). Alternatively, we can also fetch the request in a json or XML format.\n", - "\n", - "Let's first fetch the raw response and inspect it before parsing." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "816aa057-f568-48d8-9d24-e303f601c588", - "metadata": {}, - "outputs": [], - "source": [ - "def get_stations_in_bbox(minlat, maxlat, minlon, maxlon):\n", - " params = {\n", - " \"minlatitude\": minlat,\n", - " \"maxlatitude\": maxlat,\n", - " \"minlongitude\": minlon,\n", - " \"maxlongitude\": maxlon,\n", - " \"format\": \"csv\"\n", - " }\n", - " r = requests.get(f\"{BASE_URL}/gps/metadata/sites/v1\", params=params) \n", - " r.raise_for_status()\n", - " return r.text" - ] - }, - { - "cell_type": "markdown", - "id": "8d6ace4f-a738-4fa0-809b-050c034ed3f0", - "metadata": {}, - "source": [ - "`requests.get()` returns the entire HTTP response body as a single Python string. We can inspect the raw output as below:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "08af06dd-6b0d-47e4-a6e7-033963ff6b66", - "metadata": {}, - "outputs": [], - "source": [ - "meta_csv = get_stations_in_bbox(MINLAT, MAXLAT, MINLON, MAXLON)\n", - "for line in meta_csv.splitlines()[:3]:\n", - " print(line+\"\\n\")" - ] - }, - { - "cell_type": "markdown", - "id": "b85cca2a-6105-43b2-814a-c2879c751383", - "metadata": {}, - "source": [ - "The first line is the `#fields=` header that encodes column names and types. The following rows after the header row are data. Notice `CORV` appears twice with different receiver types, confirming that stations have multiple session records.\n" - ] - }, - { - "cell_type": "markdown", - "id": "8f5f4fe3-4ee3-44aa-85c5-2ee0e71a5450", - "metadata": {}, - "source": [ - "## Parsing Station Metadata" - ] - }, - { - "cell_type": "markdown", - "id": "fff13051-7723-42e0-9709-e7e6955cf0a0", - "metadata": {}, - "source": [ - "The raw GeoCSV response contains column names encoded in the `#fields=` comment line with type annotations like `[type='string']` that we need to strip out. Each station also appears multiple times in the response — once per equipment configuration (antenna/receiver changes over time). We deduplicate by station ID for getting the station list and optionally filter by our time range of interest." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "576a0863-a9bc-4f82-ba10-1274cd9c9394", - "metadata": {}, - "outputs": [], - "source": [ - "def parse_station_metadata(csv_text, start=None, end=None):\n", - " # extract column names from the #fields= comment line\n", - " fields_line = [l for l in csv_text.splitlines() if l.startswith('#fields=')][0]\n", - " col_names = [f.split('[')[0] for f in fields_line.replace('#fields=', '').split(',')]\n", - "\n", - " # parse data rows (skip all comment lines)\n", - " lines = [l for l in csv_text.splitlines() if not l.startswith('#')]\n", - " df = pd.read_csv(StringIO(\"\\n\".join(lines)), header=None)\n", - " df.columns = col_names\n", - "\n", - " # convert session times to datetime\n", - " df[\"session_start_time\"] = pd.to_datetime(df[\"session_start_time\"])\n", - " df[\"session_stop_time\"] = pd.to_datetime(df[\"session_stop_time\"])\n", - "\n", - " # filter to stations active during our time range\n", - " # a station is included if it has any overlap with [start, end]\n", - " if start:\n", - " df = df[df[\"session_stop_time\"] >= pd.to_datetime(start, utc=True)]\n", - " if end:\n", - " df = df[df[\"session_start_time\"] <= pd.to_datetime(end, utc=True)]\n", - "\n", - " # one row per station\n", - " df = df.drop_duplicates(subset=\"ID\")\n", - "\n", - " return df[[\"ID\", \"latitude\", \"longitude\"]]" - ] - }, - { - "cell_type": "markdown", - "id": "32b41240-6047-4856-b47c-0a244bc4c4d4", - "metadata": {}, - "source": [ - "We now parse the raw response and filter to stations that have data during our time range. The result is one row per station with its ID and coordinates." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "65b95d76-9e5d-4358-80a1-54f537576a92", - "metadata": {}, - "outputs": [], - "source": [ - "stations = parse_station_metadata(meta_csv, start=START, end=END)\n", - "print(f\"Found {len(stations)} stations\")\n", - "stations.reset_index(drop=True).head()" - ] - }, - { - "cell_type": "markdown", - "id": "e1ea2e50-f4fb-44f2-8e63-6d056a5fa33d", - "metadata": {}, - "source": [ - "## Visualizing Stations on a Map" - ] - }, - { - "cell_type": "markdown", - "id": "ddcf39c7-b153-4db2-8a18-4aed87ef9c6f", - "metadata": {}, - "source": [ - "Now that we have our station list, we can add them to the map alongside our bounding box to give us a complete picture of our study area i.e. where the bbox is geographically and which stations fall within it." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "a0f8de95-cb87-46f3-9271-c8595e19c4ed", - "metadata": {}, - "outputs": [], - "source": [ - "def plot_stations_map(stations_df, minlat, maxlat, minlon, maxlon, height=500):\n", - " center_lat = (minlat + maxlat) / 2\n", - " center_lon = (minlon + maxlon) / 2\n", - "\n", - " fig = folium.Figure(height=height)\n", - " m = folium.Map(location=[center_lat, center_lon], zoom_start=7,\n", - " tiles=\"CartoDB positron\")\n", - " m.add_to(fig)\n", - "\n", - " # bounding box\n", - " folium.Rectangle(\n", - " bounds=[[minlat, minlon], [maxlat, maxlon]],\n", - " color=\"blue\",\n", - " fill=True,\n", - " fill_opacity=0.1,\n", - " tooltip=f\"Bounding box: ({minlat}, {minlon}) to ({maxlat}, {maxlon})\"\n", - " ).add_to(m)\n", - "\n", - " # station markers\n", - " for _, row in stations_df.iterrows():\n", - " folium.RegularPolygonMarker(\n", - " location=[row[\"latitude\"], row[\"longitude\"]],\n", - " number_of_sides=3,\n", - " radius=5,\n", - " rotation=30,\n", - " color=\"red\",\n", - " fill=True,\n", - " fill_opacity=1,\n", - " popup=row[\"ID\"],\n", - " tooltip=\"Station: \"+row[\"ID\"]\n", - " ).add_to(m)\n", - "\n", - " return fig" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "eeb2a430-7877-43c9-b79d-9bff44de03fa", - "metadata": {}, - "outputs": [], - "source": [ - "plot_stations_map(stations, MINLAT, MAXLAT, MINLON, MAXLON)" - ] - }, - { - "cell_type": "markdown", - "id": "7d43dfc4-b2dc-4a71-acc9-77bfb5c3e014", - "metadata": {}, - "source": [ - "You should see red triangle markers within the blue bounding box. Click or hover over a marker to see the station ID. If no stations appear, your bounding box may be too small or your date range too restrictive." - ] - }, - { - "cell_type": "markdown", - "id": "f62e9029-7600-4bb3-857e-542413277761", - "metadata": {}, - "source": [ - "## Retrieving Position Time Series" - ] - }, - { - "cell_type": "markdown", - "id": "55c3afc2-545a-4fa9-b232-395bc8b79abc", - "metadata": {}, - "source": [ - "Now that we have our station list and have verified the locations on the map, we can fetch the processed position time series for each station from the GAGE API endpoint `/gps/data/position/{station}/v3`.\n", - "\n", - "The API returns a response with north, east, and up displacement offsets relative to a reference coordinate at each station. The API parameters we use are:\n", - "\n", - "- **analysisCenter:** GNSS time series position solutions are available from four different analysis centers. CWU (default), NMT, PBO and UNR.\n", - "- **referenceFrame:** The position solutions are available in different reference frames depending on the analysis center. `nam14` (North America fixed) is the default whereas the global frame `igs14` is available from all analysis centers.\n", - "- **report:** controls the output content:\n", - " - `short` - (default) north/east/up offsets and standard deviations only\n", - " - `long` - full source file including cartesian and geodetic coordinates and all associated error estimates and covariances\n", - "- **dataPostProcessing:** post-processing applied after data retrieval:\n", - " - `Uncleaned` (default) — data returned as-is, no post-processing\n", - " - `Cleaned` — offset values for North, East, or Up are set to NULL when the standard deviation exceeds 20mm\n", - "- **format:** Response formats. CSV, JSON or XML.\n", - "\n", - "Data is always fetched fresh from the API and saved to your scratch storage." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "17e7b84c-4ffa-407a-9ed1-5676b7280f65", - "metadata": {}, - "outputs": [], - "source": [ - "SAVE_DIR = os.path.join(os.environ[\"SCRATCH_BUCKET\"], \"gnss_positions\")\n", - "shutil.rmtree(SAVE_DIR) if os.path.exists(SAVE_DIR) else None\n", - "\n", - "def get_position_timeseries(station_code, start=None, end=None):\n", - " params = {\n", - " \"analysisCenter\": \"cwu\",\n", - " \"referenceFrame\": \"nam14\",\n", - " \"report\": \"short\",\n", - " \"dataPostProcessing\": \"Cleaned\",\n", - " \"format\": \"csv\"\n", - " }\n", - " if start:\n", - " params[\"starttime\"] = start\n", - " if end:\n", - " params[\"endtime\"] = end\n", - "\n", - " r = requests.get(f\"{BASE_URL}/gps/data/position/{station_code}/v3\", params=params)\n", - " r.raise_for_status()\n", - " \n", - " os.makedirs(SAVE_DIR, exist_ok=True)\n", - " filepath = os.path.join(SAVE_DIR, f\"{station_code}.csv\")\n", - " with open(filepath, \"w\") as f:\n", - " f.write(r.text)\n", - " # print(f\" Saved to {filepath}\")\n", - " return r.text\n", - "\n", - "\n", - "def parse_position_csv(csv_text):\n", - " lines = [l for l in csv_text.splitlines() if not l.startswith('#')]\n", - " df = pd.read_csv(StringIO(\"\\n\".join(lines)))\n", - " df.columns = df.columns.str.strip()\n", - " df[\"Datetime\"] = pd.to_datetime(df[\"Datetime\"])\n", - " return df" - ] - }, - { - "cell_type": "markdown", - "id": "65bc6ece-a770-49a1-9963-f3fcdd854ef1", - "metadata": {}, - "source": [ - "Before fetching all stations, we test on a single station to verify the response format and confirm our parameters are working correctly." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "cea9a987-b648-487e-87bb-b44febbe9597", - "metadata": {}, - "outputs": [], - "source": [ - "# fetch and inspect a single station before running all stations\n", - "code = stations[\"ID\"].iloc[0]\n", - "print(f\"Fetching {code}...\")\n", - "pos_csv = get_position_timeseries(code, start=START, end=END)\n", - "df = parse_position_csv(pos_csv)\n", - "print(f\"{len(df)} epochs\")\n", - "df.head()" - ] - }, - { - "cell_type": "markdown", - "id": "f4568f27-e5a4-424e-991e-b6550f4efff3", - "metadata": {}, - "source": [ - "You should see a DataFrame with columns: `Datetime`, `delta N`, `delta E`, `delta U`, and associated standard deviations. Each row is one daily position estimate. " - ] - }, - { - "cell_type": "markdown", - "id": "e436d329-632a-4258-8253-65bc7bd7201b", - "metadata": {}, - "source": [ - "## Retrieving Data for All Stations" - ] - }, - { - "cell_type": "markdown", - "id": "50e7502c-fcff-46c1-a79f-c9aa72857ecc", - "metadata": {}, - "source": [ - "Now we fetch position time series for all stations. Stations that return a 404 error will be skipped automatically. See the Troubleshooting section for guidance." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "38e6a91f-c756-44e8-94b2-f6c85ff97cc8", - "metadata": {}, - "outputs": [], - "source": [ - "for _, row in stations.iterrows():\n", - " code = row[\"ID\"]\n", - " print(f\"Fetching {code}...\")\n", - " try:\n", - " pos_csv = get_position_timeseries(code, start=START, end=END)\n", - " df = parse_position_csv(pos_csv)\n", - " print(f\" {len(df)} epochs\")\n", - " except requests.HTTPError as e:\n", - " print(f\" Skipping {code}: {e}\")" - ] - }, - { - "cell_type": "markdown", - "id": "10ba2f97-b3cd-4d6c-b322-c54c8e1372d0", - "metadata": {}, - "source": [ - "## Summary\n", - "\n", - "Let's print a summary of what was retrieved to confirm everything looks as expected \n", - "before moving on to analysis." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "ed8d0758-0a62-4145-9739-0f0cca2e244d", - "metadata": {}, - "outputs": [], - "source": [ - "successful = []\n", - "failed = []\n", - "\n", - "for _, row in stations.iterrows():\n", - " code = row[\"ID\"]\n", - " filepath = os.path.join(SAVE_DIR, f\"{code}.csv\")\n", - " if os.path.exists(filepath):\n", - " successful.append(code)\n", - " else:\n", - " failed.append(code)\n", - "\n", - "print(f\"Total stations found: {len(stations)}\")\n", - "print(f\"Successfully retrieved: {len(successful)}\")\n", - "print(f\"Failed: {len(failed)}\")\n", - "\n", - "if failed:\n", - " print(f\"\\nFailed stations: {failed}\")\n", - "\n", - "print(f\"\\nData saved to: {SAVE_DIR}\")" - ] - }, - { - "cell_type": "markdown", - "id": "3db5fdc7-54c3-4588-b685-e6aeda88cf6f", - "metadata": {}, - "source": [ - "## Troubleshooting\n", - "\n", - "- If a station returns a 404 error, for example `Skipping station_ID: 404 Client Error` it exists in the metadata but has no processed position solution for your date range. This is normal and the station will be skipped automatically. One solution might be to change the analysis center, for example from `cwu` to `unr` to see if another analysis center has processed the data." - ] - }, - { - "cell_type": "markdown", - "id": "811c5c9a-beb4-4293-8a47-67a809dfb39a", - "metadata": {}, - "source": [ - "## Try It Yourself\n", - "\n", - "- Change `MINLAT`, `MAXLAT`, `MINLON`, `MAXLON` to a different region and re-run\n", - "- Change `START`, `END` to a different time range and re-run\n", - "- Change `report` from `short` to `long`. What additional columns appear?" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.13" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} From 66514bf0e1ca232a05fc73712d9f8b7a0c630db1 Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Sun, 12 Jul 2026 21:51:42 +0000 Subject: [PATCH 3/6] Minor grammar fixes --- .../NB1-access-gnss-via-SDK-checkpoint.ipynb | 22 +++++++++---------- .../NB1-access-gnss-via-SDK.ipynb | 22 +++++++++---------- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb index e3b7dd5..1d9c2a6 100644 --- a/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb +++ b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb @@ -25,7 +25,7 @@ "\n", "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", "\n", - "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to reterive GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", "\n", "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", "* instantiating the EarthScope SDK client\n", @@ -61,7 +61,7 @@ "By the end of this notebook, you will be able to:\n", "\n", "1. Instantiate EarthScope SDK client\n", - "2. Retreive and filter GNSS data from EarthScope" + "2. Retrieve and filter GNSS data from EarthScope" ] }, { @@ -86,7 +86,7 @@ "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", - "6. [Larger than Memory Requests with Query Plan](#id-6-larger-than-memory-requests-with-query-plans)\n", + "6. [Larger than Memory Requests with Query Plans](#id-6-larger-than-memory-requests-with-query-plans)\n", "7. [Exploration Exercises](#id-7-exploration-exercises)\n", "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" ] @@ -109,8 +109,8 @@ "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", "\n", "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", - "2. **Carrier phase**: It is a more precise distance measurement between the satellite and thr ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", - "3. **SNR**: Signal-to-Noise Ratio. IT is a parameter that tells how strong and clean the received signal is.\n", + "2. **Carrier phase**: It is a more precise distance measurement between the satellite and the ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", + "3. **SNR**: Signal-to-Noise Ratio. It is a parameter that tells how strong and clean the received signal is.\n", "\n", "**Raw Observations vs. derived products**\n", "\n", @@ -142,7 +142,7 @@ "id": "50263265-10c7-44c7-a302-37b658b899c8", "metadata": {}, "source": [ - "### Instantiating the Earthscope SDK" + "### Instantiating the EarthScope SDK" ] }, { @@ -341,7 +341,7 @@ "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", "\n", - "Fpr example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." + "For example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." ] }, { @@ -411,7 +411,7 @@ "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", "metadata": {}, "source": [ - "## 6. Larger than Memory Requests with Query Plan" + "## 6. Larger than Memory Requests with Query Plans" ] }, { @@ -427,7 +427,7 @@ "id": "7a61354a-548f-4459-976c-89d94b1d5104", "metadata": {}, "source": [ - "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageble groups such as by day or by station. This allows you to proccess one group at a time.\n", + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageable groups such as by day or by station. This allows you to process one group at a time.\n", "\n", "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", "\n", @@ -517,7 +517,7 @@ "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", "metadata": {}, "source": [ - " > check: Each line of output represents one group." + " > **Check**: Each line of output represents one group." ] }, { @@ -543,7 +543,7 @@ "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", "metadata": {}, "source": [ - "For cutom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [Earthscope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." + "For custom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [EarthScope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." ] }, { diff --git a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb index e3b7dd5..1d9c2a6 100644 --- a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb +++ b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb @@ -25,7 +25,7 @@ "\n", "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", "\n", - "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to reterive GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", "\n", "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", "* instantiating the EarthScope SDK client\n", @@ -61,7 +61,7 @@ "By the end of this notebook, you will be able to:\n", "\n", "1. Instantiate EarthScope SDK client\n", - "2. Retreive and filter GNSS data from EarthScope" + "2. Retrieve and filter GNSS data from EarthScope" ] }, { @@ -86,7 +86,7 @@ "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", - "6. [Larger than Memory Requests with Query Plan](#id-6-larger-than-memory-requests-with-query-plans)\n", + "6. [Larger than Memory Requests with Query Plans](#id-6-larger-than-memory-requests-with-query-plans)\n", "7. [Exploration Exercises](#id-7-exploration-exercises)\n", "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" ] @@ -109,8 +109,8 @@ "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", "\n", "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", - "2. **Carrier phase**: It is a more precise distance measurement between the satellite and thr ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", - "3. **SNR**: Signal-to-Noise Ratio. IT is a parameter that tells how strong and clean the received signal is.\n", + "2. **Carrier phase**: It is a more precise distance measurement between the satellite and the ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", + "3. **SNR**: Signal-to-Noise Ratio. It is a parameter that tells how strong and clean the received signal is.\n", "\n", "**Raw Observations vs. derived products**\n", "\n", @@ -142,7 +142,7 @@ "id": "50263265-10c7-44c7-a302-37b658b899c8", "metadata": {}, "source": [ - "### Instantiating the Earthscope SDK" + "### Instantiating the EarthScope SDK" ] }, { @@ -341,7 +341,7 @@ "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", "\n", - "Fpr example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." + "For example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." ] }, { @@ -411,7 +411,7 @@ "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", "metadata": {}, "source": [ - "## 6. Larger than Memory Requests with Query Plan" + "## 6. Larger than Memory Requests with Query Plans" ] }, { @@ -427,7 +427,7 @@ "id": "7a61354a-548f-4459-976c-89d94b1d5104", "metadata": {}, "source": [ - "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageble groups such as by day or by station. This allows you to proccess one group at a time.\n", + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageable groups such as by day or by station. This allows you to process one group at a time.\n", "\n", "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", "\n", @@ -517,7 +517,7 @@ "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", "metadata": {}, "source": [ - " > check: Each line of output represents one group." + " > **Check**: Each line of output represents one group." ] }, { @@ -543,7 +543,7 @@ "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", "metadata": {}, "source": [ - "For cutom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [Earthscope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." + "For custom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [EarthScope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." ] }, { From 5d95d5c1e46a74bc8db284f54ae7bf5d8476c571 Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Mon, 13 Jul 2026 21:36:11 +0000 Subject: [PATCH 4/6] add first geodesy notebook that uses EarthScope SDK --- .../NB1-access-gnss-via-SDK.ipynb | 26 +++++++++---------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb index 1d9c2a6..a69fe75 100644 --- a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb +++ b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb @@ -23,7 +23,7 @@ "source": [ "## Introduction\n", "\n", - "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", + "**What this notebook does:** It instantiates the EarthScope SDK and allows you to retrieve GNSS observations for your station of interest.\n", "\n", "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", "\n", @@ -106,7 +106,7 @@ "source": [ "A GNSS (Global Navigation Satellite System) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", "\n", - "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", + "Each of those recordings is a GNSS observation. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", "\n", "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", "2. **Carrier phase**: It is a more precise distance measurement between the satellite and the ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", @@ -114,7 +114,7 @@ "\n", "**Raw Observations vs. derived products**\n", "\n", - "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " + "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such as a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " ] }, { @@ -225,11 +225,11 @@ "id": "af77148f-64a0-424a-9dc2-2ec34a8f96e8", "metadata": {}, "source": [ - "In this step, we will retrieve GNSS observations for a single station and time window as an Apache arrow table through the EarthScope data API.\n", + "In this step, we will retrieve GNSS observations for a single station and time window as an Apache Arrow table through the EarthScope data API.\n", "\n", "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", "\n", - "The Expected output is a single table with one row per satellite, per signal, per epoch" + "The expected output is a single table with one row per satellite, per signal, per epoch" ] }, { @@ -239,7 +239,7 @@ "metadata": {}, "outputs": [], "source": [ - "# Step: describe the request, then .fetch() to run the query.\n", + "#describe the request, then .fetch() to run the query.\n", "table = await es.data.gnss_observations(\n", " start_datetime=START,\n", " end_datetime=END,\n", @@ -294,11 +294,9 @@ "outputs": [], "source": [ "# Inspect structure and coverage\n", - "print(df.schema) # column names and types\n", - "print(\"Constellations:\", df[\"system\"].unique().to_list())\n", - "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())\n", - "\n", - "df.describe() # summary statistics" + "print(df.schema) \n", + "print(\"Constellations:\", df[\"system\"].unique().sort().to_list())\n", + "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())" ] }, { @@ -312,7 +310,7 @@ "|---|---|---|\n", "| `timestamp` | datetime (UTC) | The epoch of the observation. |\n", "| `satellite` | int | Satellite number within its constellation (its PRN / slot). Combine with `system` for a globally unique ID. |\n", - "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L`, `5Q` (decoded below). |\n", + "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L` (decoded below). |\n", "| `range` | float | Pseudorange in meters. It is the apparent satellite to receiver distance. |\n", "| `phase` | float | Carrier phase in cycles. It is a precise but ambiguous range measurement. |\n", "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher is cleaner. |\n", @@ -453,7 +451,7 @@ " system=\"G\",\n", " field=[\"phase\", \"range\", \"snr\"],\n", ")\n", - "\n", + "await plan.plan()\n", "print(plan) # preview the plan's request/group counts before any data is fetched\n", "\n", "\n", @@ -559,7 +557,7 @@ "\n", "1. **Change the station:** Set `STATION` in the Configuration section to a different 4-character station ID and re-run Section 3. Does the station report the same constellations?\n", "\n", - "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"5Q\"`) with `field=\"snr\"`. How much smaller is the result?\n", + "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"2L\"`) with `field=\"snr\"`. How much smaller is the result?\n", "\n", "3. **Save your results:** Write a fetched dataframe to the scratch directory (*Hint: Use `os.environ[\"SCRATCH_BUCKET\"]`*) as Parquet with `df.write_parquet(...)`, then read it back. Parquet preserves types and is far smaller than CSV." ] From 9cd1109bcad93b2b40ec9a428782185791c1f9f5 Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Thu, 6 Aug 2026 14:34:41 +0000 Subject: [PATCH 5/6] Address review comments & remove checkpoint files --- .../NB1-access-gnss-via-SDK-checkpoint.ipynb | 614 ------------------ .../NB1-access-gnss-via-SDK.ipynb | 95 ++- 2 files changed, 69 insertions(+), 640 deletions(-) delete mode 100644 tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb diff --git a/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb deleted file mode 100644 index 1d9c2a6..0000000 --- a/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb +++ /dev/null @@ -1,614 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "2abe5d1d-4a2f-4c96-a78c-6c3e5f9ac750", - "metadata": {}, - "source": [ - "# Accessing GNSS Observations with the EarthScope SDK" - ] - }, - { - "cell_type": "markdown", - "id": "35320cb4-aed2-4420-8f9c-b3deb365e792", - "metadata": {}, - "source": [ - "**Version:** 1.0 | **Last updated:** 2026-07-09 | **Author:** Eshanta Mishra" - ] - }, - { - "cell_type": "markdown", - "id": "b9b1d1c3-569f-459a-8595-e1ba758e4bc7", - "metadata": {}, - "source": [ - "## Introduction\n", - "\n", - "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", - "\n", - "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", - "\n", - "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", - "* instantiating the EarthScope SDK client\n", - "* Select data for stations using the station name and time range.\n", - "* Retrieve observations and load into dataframes.\n", - "* Filter (slice) the GNSS data based on different parameters.\n", - "\n", - "---\n", - "\n", - "### Prerequisites\n", - "\n", - "\n", - "Before starting this notebook, you should:\n", - "* [ ] This is an introductory notebook. Being familiar with **Geodesy, GNSS, python dataframes** is recommended but not required.\n", - "\n", - "---\n", - "\n", - "### GeoLab Compute Resources\n", - "\n", - "| Setting | Recommended |\n", - "|---|---|\n", - "| **Image** | GeoLab (default image)|\n", - "| **Server size** | 4 GB RAM, ~0.5 CPUs (default server) |" - ] - }, - { - "cell_type": "markdown", - "id": "811fc68e-1e81-482c-82dd-5efa2a89d715", - "metadata": {}, - "source": [ - "## Learning Objectives\n", - "\n", - "By the end of this notebook, you will be able to:\n", - "\n", - "1. Instantiate EarthScope SDK client\n", - "2. Retrieve and filter GNSS data from EarthScope" - ] - }, - { - "cell_type": "markdown", - "id": "1ec3d79a-ad70-4c58-b8fc-287501adb7e9", - "metadata": {}, - "source": [ - "## Relevant Documentation & Resources\n", - "\n", - "* [EarthScope SDK documentation](https://docs.earthscope.org/sdk)" - ] - }, - { - "cell_type": "markdown", - "id": "076d400d-8c88-4113-a28d-91dac8648679", - "metadata": {}, - "source": [ - "## Contents\n", - "\n", - "1. [Basics of GNSS](#id-1-basics-of-gnss)\n", - "2. [Setup & Imports](#id-2-setup-imports)\n", - "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", - "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", - "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", - "6. [Larger than Memory Requests with Query Plans](#id-6-larger-than-memory-requests-with-query-plans)\n", - "7. [Exploration Exercises](#id-7-exploration-exercises)\n", - "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" - ] - }, - { - "cell_type": "markdown", - "id": "1b245a5d-5792-461d-bd51-99e55ab797a0", - "metadata": {}, - "source": [ - "## 1. Basics of GNSS" - ] - }, - { - "cell_type": "markdown", - "id": "9e03f7ee-00b4-4501-bd77-551d49ca52a6", - "metadata": {}, - "source": [ - "A GNSS (Global Navigation Satellite System) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", - "\n", - "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", - "\n", - "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", - "2. **Carrier phase**: It is a more precise distance measurement between the satellite and the ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", - "3. **SNR**: Signal-to-Noise Ratio. It is a parameter that tells how strong and clean the received signal is.\n", - "\n", - "**Raw Observations vs. derived products**\n", - "\n", - "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " - ] - }, - { - "cell_type": "markdown", - "id": "cca79b91-0348-4ee2-842e-740595288820", - "metadata": {}, - "source": [ - "## 2. Setup & Imports" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "8cd2903c-1b18-4877-a421-f19374f8fcf3", - "metadata": {}, - "outputs": [], - "source": [ - "import datetime as dt\n", - "import polars as pl\n", - "from earthscope_sdk import AsyncEarthScopeClient" - ] - }, - { - "cell_type": "markdown", - "id": "50263265-10c7-44c7-a302-37b658b899c8", - "metadata": {}, - "source": [ - "### Instantiating the EarthScope SDK" - ] - }, - { - "cell_type": "markdown", - "id": "8683b5f9-3ffc-4eb4-a8ec-82c61fa196e7", - "metadata": {}, - "source": [ - "#### Authentication" - ] - }, - { - "cell_type": "markdown", - "id": "52c5e185-8dca-42ff-8a9d-89c23297a761", - "metadata": {}, - "source": [ - "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are always available inside it. So, the client authenticates automatically. This removes the step of logging in or passing tokens manually, as you would do in a non GeoLab Environment." - ] - }, - { - "cell_type": "markdown", - "id": "b0143b85-f435-4ce2-9359-6b5f2b8d3674", - "metadata": {}, - "source": [ - "#### Async client" - ] - }, - { - "cell_type": "markdown", - "id": "421226b8-42bc-4d91-a517-469c29fa6e85", - "metadata": {}, - "source": [ - "`AsyncEarthScopeClient` is the asynchronous client. Async lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "abe833e0-9ca9-48bd-8bfc-749579592f92", - "metadata": {}, - "outputs": [], - "source": [ - "es = AsyncEarthScopeClient()" - ] - }, - { - "cell_type": "markdown", - "id": "aa3e8c77-ceaf-4f6e-be2e-b99b0b7f0125", - "metadata": {}, - "source": [ - "### Configuration\n", - "\n", - "Set your parameters here before running the rest of the notebook. Every subsequent cells read from these variables. So, this is the only place you need to edit to point the notebook at different data." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "15ecb388-d718-4949-a072-5db6cd398d65", - "metadata": {}, - "outputs": [], - "source": [ - "# Modify these values before running the notebook.\n", - "\n", - "STATION = \"AC60\" # GNSS station (4-character ID)\n", - "SESSION = \"A\" # Session name\n", - "START = dt.datetime(2025, 7, 20, 21) # Query start (UTC)\n", - "END = dt.datetime(2025, 7, 21, 3) # Query end (UTC)" - ] - }, - { - "cell_type": "markdown", - "id": "f2e9e575-a7b1-4dd8-86a9-17f0fd942f15", - "metadata": {}, - "source": [ - "## 3. Retrieve Observations for a single station" - ] - }, - { - "cell_type": "markdown", - "id": "af77148f-64a0-424a-9dc2-2ec34a8f96e8", - "metadata": {}, - "source": [ - "In this step, we will retrieve GNSS observations for a single station and time window as an Apache arrow table through the EarthScope data API.\n", - "\n", - "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", - "\n", - "The Expected output is a single table with one row per satellite, per signal, per epoch" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "4eb07e18-b72b-4ed4-99b7-558d935fee31", - "metadata": {}, - "outputs": [], - "source": [ - "# Step: describe the request, then .fetch() to run the query.\n", - "table = await es.data.gnss_observations(\n", - " start_datetime=START,\n", - " end_datetime=END,\n", - " station_name=STATION,\n", - " session_name=SESSION,\n", - ").fetch()\n", - "\n", - "table" - ] - }, - { - "cell_type": "markdown", - "id": "b5d3fab2-0ab0-4231-9681-b6e56a117f9d", - "metadata": {}, - "source": [ - "Convert the Arrow table to a Polars dataframe with `pl.from_arrow(...)`. This is zero-copy, so it is very efficient. Sorting by `timestamp` makes the rows read chronologically." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "82b54655-559f-4b95-8479-a386affe5b91", - "metadata": {}, - "outputs": [], - "source": [ - "df = pl.from_arrow(table).sort(\"timestamp\")\n", - "df" - ] - }, - { - "cell_type": "markdown", - "id": "05678a5b-388b-4d44-8e04-28a8912af92e", - "metadata": {}, - "source": [ - "## 4. Understanding the Observation fields" - ] - }, - { - "cell_type": "markdown", - "id": "6f775aef-3b48-4a78-9d4f-b43b7cd1c143", - "metadata": {}, - "source": [ - "The default data query returns every field - the same information you would find in a RINEX observation file. Let us take a moment to understand the data structure.\n", - "Each row of data is a single measurement (one satellite, one signal, one instant). Inspect the columns, the constellations present, and the range of signal strengths." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1ab79788-7803-4711-aa4f-f977b8e83e7a", - "metadata": {}, - "outputs": [], - "source": [ - "# Inspect structure and coverage\n", - "print(df.schema) # column names and types\n", - "print(\"Constellations:\", df[\"system\"].unique().to_list())\n", - "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())\n", - "\n", - "df.describe() # summary statistics" - ] - }, - { - "cell_type": "markdown", - "id": "c59fb35b-ec60-4879-8a0e-1c6bbe9e4a4a", - "metadata": {}, - "source": [ - "### What each field means\n", - "\n", - "| Column | Type | Meaning |\n", - "|---|---|---|\n", - "| `timestamp` | datetime (UTC) | The epoch of the observation. |\n", - "| `satellite` | int | Satellite number within its constellation (its PRN / slot). Combine with `system` for a globally unique ID. |\n", - "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L`, `5Q` (decoded below). |\n", - "| `range` | float | Pseudorange in meters. It is the apparent satellite to receiver distance. |\n", - "| `phase` | float | Carrier phase in cycles. It is a precise but ambiguous range measurement. |\n", - "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher is cleaner. |\n", - "| `slip` | int | Cycle-slip indicator; `null` when phase tracking was continuous. |\n", - "| `flags` | int | Per-observation status flags (e.g. loss-of-lock). |\n", - "| `fcn` | int | Frequency channel number — used by GLONASS's FDMA signals; `0` otherwise. |\n", - "| `system` | str | Constellation code (see below). |\n", - "| `igs` | str | The station's IGS long name (e.g. `AC6000USA`). |\n", - "\n", - "### Reading the `system` column\n", - "\n", - "| Code | Constellation |\n", - "|---|---|\n", - "| `G` | GPS (USA) |\n", - "| `R` | GLONASS (Russia) |\n", - "| `E` | Galileo (EU) |\n", - "| `C` | BeiDou (China) |\n", - "| `J` | QZSS (Japan) |\n", - "| `I` | NavIC / IRNSS (India) |\n", - "| `S` | SBAS (augmentation) |\n", - "\n", - "### Reading the `obs_code` column\n", - "\n", - "An observation code names a specific signal as **band + tracking attribute**:\n", - "\n", - "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", - "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", - "\n", - "For example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." - ] - }, - { - "cell_type": "markdown", - "id": "7f1cd227-e581-4790-97a7-176bfc846f2a", - "metadata": {}, - "source": [ - "## 5. Requesting Only the Data You Need" - ] - }, - { - "cell_type": "markdown", - "id": "c0a4e066-7b6b-4161-9260-a4a825637f2a", - "metadata": {}, - "source": [ - "When you are working with RINEX files, the files hand you everything, all fields, constellations, satellites etc. Most analyses only need a small slice of these data. Passing filters to `gnss_observations()` in the SDK pushes that selection to the server, so only the data that you asked for is transferred.\n", - "\n", - "Downloading a single signal from a single satellite over two months is drastically cheaper than downloading full RINEX and discarding 99% of it. Less data means faster fetches and smaller memory footprints.\n", - "\n", - "The resulting output is a dataframe containing only the requested stations, constellations, satellites, signals and fields. For example, only `snr` and `range` columns when `field=[\"snr\",\"range\"]`.\n", - "\n", - "Each filter below is optional, accepts a single value or a list, and shrinks the request:\n", - "\n", - "* `station_name`: one name or a list of names\n", - "* `network_name`: request a whole network at once\n", - "* `system`: constellation code(s), e.g. `\"G\"` or `[\"G\", \"R\"]`\n", - "* `satellite`: specific satellite number(s)\n", - "* `obs_code`: specific signal(s), e.g. `[\"1C\", \"2L\"]`\n", - "* `field`: which measurement column(s) to return, e.g. `[\"snr\", \"range\"]`\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "949a6db5-f949-43f9-930c-88344765e60b", - "metadata": {}, - "outputs": [], - "source": [ - "start = dt.datetime(2025, 7, 20)\n", - "end = start + dt.timedelta(days=7)\n", - "\n", - "table = await es.data.gnss_observations(\n", - " start_datetime=start,\n", - " end_datetime=end,\n", - " station_name=[\"P717\", \"P453\", \"P146\", \"P147\", \"P041\"],\n", - " session_name=\"A\",\n", - " system=[\"G\", \"R\"],\n", - " obs_code=[\"1C\", \"2L\"],\n", - " satellite=[\"7\", \"21\", \"28\"],\n", - " field=[\"snr\", \"range\"],\n", - ").fetch()\n", - "\n", - "df_sliced = pl.from_arrow(table).sort(\"timestamp\")\n", - "df_sliced" - ] - }, - { - "cell_type": "markdown", - "id": "0cad076e-a951-416e-ad88-96dd7fe1f253", - "metadata": {}, - "source": [ - "> **Check:** The result should contain only the `snr` and `range` measurement columns, no `phase`, no `flags`." - ] - }, - { - "cell_type": "markdown", - "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", - "metadata": {}, - "source": [ - "## 6. Larger than Memory Requests with Query Plans" - ] - }, - { - "cell_type": "markdown", - "id": "81e989fa-baf0-4396-920e-fece9f620f43", - "metadata": {}, - "source": [ - "### What is a query plan?" - ] - }, - { - "cell_type": "markdown", - "id": "7a61354a-548f-4459-976c-89d94b1d5104", - "metadata": {}, - "source": [ - "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageable groups such as by day or by station. This allows you to process one group at a time.\n", - "\n", - "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", - "\n", - "The expected output is a summary for each group (row count, time span, and stations), rather than one huge dataframe." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1772dd83-792b-4666-a107-392d8522b92f", - "metadata": {}, - "outputs": [], - "source": [ - "# Build a query plan (note: no .fetch())\n", - "start = dt.datetime(2025, 7, 20)\n", - "end = start + dt.timedelta(days=7)\n", - "\n", - "plan = es.data.gnss_observations(\n", - " start_datetime=start,\n", - " end_datetime=end,\n", - " network_name=\"PERM:Alaska\",\n", - " session_name=\"A\",\n", - " system=\"G\",\n", - " field=[\"phase\", \"range\", \"snr\"],\n", - ")\n", - "\n", - "print(plan) # preview the plan's request/group counts before any data is fetched\n", - "\n", - "\n", - "def summarize(table):\n", - " # Print a quick summary of one group's table.\n", - " d = pl.from_arrow(table)\n", - " stations = d[\"igs\"].unique().sort().to_list()\n", - " if len(stations) > 10:\n", - " stations = f\"{len(stations)} stations\"\n", - " print(len(d), d[\"timestamp\"].min(), d[\"timestamp\"].max(), stations)" - ] - }, - { - "cell_type": "markdown", - "id": "6f59fcbd-b77a-4733-82cc-b064b081bc66", - "metadata": {}, - "source": [ - "Printing a plan reports how many API requests and groups it will run before any data moves. This provides the users with a quick way to gauge how large a query is." - ] - }, - { - "cell_type": "markdown", - "id": "ac24a294-a48d-416a-a285-a2abd1f633c5", - "metadata": {}, - "source": [ - "Processing the plan one day at a time. You never hold more than a single day in memory at once." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "46d1af54-622f-4066-9895-cfd595341cdd", - "metadata": {}, - "outputs": [], - "source": [ - "async for table in plan.group_by_day():\n", - " summarize(table)" - ] - }, - { - "cell_type": "markdown", - "id": "02c76245-1e7a-4663-b398-f0472a58eb68", - "metadata": {}, - "source": [ - "Processing the plan one station at a time across the whole window instead." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "34742255-bd16-4197-b151-1777b4753a1d", - "metadata": {}, - "outputs": [], - "source": [ - "async for table in plan.group_by_station():\n", - " summarize(table)" - ] - }, - { - "cell_type": "markdown", - "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", - "metadata": {}, - "source": [ - " > **Check**: Each line of output represents one group." - ] - }, - { - "cell_type": "markdown", - "id": "8e88b724-65ec-4ef7-9e19-fb6bbf8c895b", - "metadata": {}, - "source": [ - "### A rough guide for choosing a strategy" - ] - }, - { - "cell_type": "markdown", - "id": "d0673a80-e5e9-4256-8f3f-a48ba2ddafa2", - "metadata": {}, - "source": [ - "* Less than a week: `.fetch()` the whole results at once.\n", - "* Weeks to months, or many stations: iterate with `group_by_day()` (or `group_by_station()`)\n", - "* Months to years: define custom batches with `plan.group_by()`." - ] - }, - { - "cell_type": "markdown", - "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", - "metadata": {}, - "source": [ - "For custom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [EarthScope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." - ] - }, - { - "cell_type": "markdown", - "id": "c5008800-7811-4fbc-9721-9869da927fc2", - "metadata": {}, - "source": [ - "## 7. Exploration Exercises\n", - "\n", - "Now that you've completed the core workflow, try modifying the parameters below to explore how the results change.\n", - "\n", - "**Try these modifications:**\n", - "\n", - "1. **Change the station:** Set `STATION` in the Configuration section to a different 4-character station ID and re-run Section 3. Does the station report the same constellations?\n", - "\n", - "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"5Q\"`) with `field=\"snr\"`. How much smaller is the result?\n", - "\n", - "3. **Save your results:** Write a fetched dataframe to the scratch directory (*Hint: Use `os.environ[\"SCRATCH_BUCKET\"]`*) as Parquet with `df.write_parquet(...)`, then read it back. Parquet preserves types and is far smaller than CSV." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "edf9b94a-b045-4fcf-be34-f2e2f958f124", - "metadata": {}, - "outputs": [], - "source": [ - "# Exploration cell — use this space to experiment" - ] - }, - { - "cell_type": "markdown", - "id": "1e147f5a-2352-4a16-b28b-3f5c1dbea902", - "metadata": {}, - "source": [ - "## 8. Troubleshooting & Support\n", - "\n", - "### Further Resources\n", - "\n", - "* [EarthScope SDK Documentation](https://docs.earthscope.org/sdk)\n", - "* [SDK GNSS Observation tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", - "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", - "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.13" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb index a69fe75..14926f6 100644 --- a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb +++ b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb @@ -13,7 +13,14 @@ "id": "35320cb4-aed2-4420-8f9c-b3deb365e792", "metadata": {}, "source": [ - "**Version:** 1.0 | **Last updated:** 2026-07-09 | **Author:** Eshanta Mishra" + "**Version:** 1.0 | **Last updated:** 2026-07-09 \n", + "**Author:** Eshanta Mishra | **Author institution:** EarthScope Consortium\n", + "\n", + "**Maintainer:** EarthScope OnRamp Team | **Maintainer's contact :** help@earthscope.org\n", + "\n", + "**Estimated Time:** ~ 30 minutes| **Pathway:** MVP1\n", + "\n", + "**License:** CC-BY-4.0" ] }, { @@ -114,7 +121,7 @@ "\n", "**Raw Observations vs. derived products**\n", "\n", - "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such as a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " + "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such as a station's position, displacement over time, etc., are derived products. These derived products are computed by processing many observations together. " ] }, { @@ -134,6 +141,8 @@ "source": [ "import datetime as dt\n", "import polars as pl\n", + "import os\n", + "\n", "from earthscope_sdk import AsyncEarthScopeClient" ] }, @@ -158,7 +167,7 @@ "id": "52c5e185-8dca-42ff-8a9d-89c23297a761", "metadata": {}, "source": [ - "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are always available inside it. So, the client authenticates automatically. This removes the step of logging in or passing tokens manually, as you would do in a non GeoLab Environment." + "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are always available inside it. So, the client authenticates automatically. This removes the step of logging in or passing tokens manually, as you would do in a non GeoLab Environment. However, if you are running this notebook outside geolab, you will need to authenticate manually using your EarthScope account credentials through the EarthScope CLI (Detailed instructions for doing this can be found [here](https://gitlab.com/earthscope/public/earthscope-cli))." ] }, { @@ -174,7 +183,7 @@ "id": "421226b8-42bc-4d91-a517-469c29fa6e85", "metadata": {}, "source": [ - "`AsyncEarthScopeClient` is the asynchronous client. Async lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer." + "`AsyncEarthScopeClient` is the asynchronous client. [Async](https://docs.earthscope.org/sdk/usage#async-usage) lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer." ] }, { @@ -209,7 +218,11 @@ "STATION = \"AC60\" # GNSS station (4-character ID)\n", "SESSION = \"A\" # Session name\n", "START = dt.datetime(2025, 7, 20, 21) # Query start (UTC)\n", - "END = dt.datetime(2025, 7, 21, 3) # Query end (UTC)" + "END = dt.datetime(2025, 7, 21, 3) # Query end (UTC)\n", + "\n", + "OUTPUT_DIR = os.path.join(os.environ[\"SCRATCH_BUCKET\"], \"mvp1-geodesy-nb1-output\") # output folder on personal scratch space on S3\n", + "\n", + "print(f\"Configuration set. Outputs will be saved to: {OUTPUT_DIR}\")" ] }, { @@ -225,7 +238,7 @@ "id": "af77148f-64a0-424a-9dc2-2ec34a8f96e8", "metadata": {}, "source": [ - "In this step, we will retrieve GNSS observations for a single station and time window as an Apache Arrow table through the EarthScope data API.\n", + "In this step, we will retrieve GNSS observations for a single station and time window as an Apache Arrow table through the [EarthScope API](https://api.earthscope.org/beta/docs#get-/data-products/gnss/observations).\n", "\n", "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", "\n", @@ -357,7 +370,9 @@ "source": [ "When you are working with RINEX files, the files hand you everything, all fields, constellations, satellites etc. Most analyses only need a small slice of these data. Passing filters to `gnss_observations()` in the SDK pushes that selection to the server, so only the data that you asked for is transferred.\n", "\n", - "Downloading a single signal from a single satellite over two months is drastically cheaper than downloading full RINEX and discarding 99% of it. Less data means faster fetches and smaller memory footprints.\n", + "The payoff is immediate: requesting a single signal from a single satellite over two months returns in a fraction of the time and requires smaller memory size, where full RINEX for the same window would be orders of magnitude larger and mostly discarded. You also skip RINEX parsing entirely because the results arrive as an Arrow table ready to load straight into a dataframe, rather than a set of daily text files you have to decode first.\n", + "\n", + "*A smaller slice also means less data moved across the network, which keeps costs down on EarthScope's side.*\n", "\n", "The resulting output is a dataframe containing only the requested stations, constellations, satellites, signals and fields. For example, only `snr` and `range` columns when `field=[\"snr\",\"range\"]`.\n", "\n", @@ -398,10 +413,34 @@ }, { "cell_type": "markdown", - "id": "0cad076e-a951-416e-ad88-96dd7fe1f253", + "id": "7ccb66b6-ca6e-4d4a-876a-da3b40fb0361", + "metadata": {}, + "source": [ + "### Saving your results\n", + "\n", + "Fetching takes time, so it's worth writing results to disk once you have data you'll reuse. In GeoLab, where you write matters:\n", + "\n", + "| Location | Path | Persistence | Use it for |\n", + "|---|---|---|---|\n", + "| **Home** | `/home/jovyan/` | Private, persistent, 50 GB limit | Notebooks, scripts, and results you want to keep while working on your project |\n", + "| **Shared** | `/home/jovyan/shared/` | Read-only | Datasets and starter notebooks placed there by instructors. You can copy files out, but not modify them |\n", + "| **Scratch** | `/earthscope-scratch/` (exact path in `os.environ[\"SCRATCH_BUCKET\"]`) | Temporary and auto-deletes after ~2 weeks | Large intermediates over the 50 GB home limit |\n", + "\n", + "Use Parquet rather than CSV. It's columnar, compressed, and preserves data types — your `timestamp` comes back as a datetime and `snr` as a float, with no re-parsing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e9c627a6-7f55-46bc-98bf-abd26db7b3dd", "metadata": {}, + "outputs": [], "source": [ - "> **Check:** The result should contain only the `snr` and `range` measurement columns, no `phase`, no `flags`." + "# Step description: Save the sliced dataframe to your Scratch directory.\n", + "out_path = os.path.join(OUTPUT_DIR, \"sliced_observations.parquet\")\n", + "df_sliced.write_parquet(out_path)\n", + "\n", + "print(f\"Wrote {len(df_sliced):,} rows to {out_path}\")" ] }, { @@ -425,7 +464,7 @@ "id": "7a61354a-548f-4459-976c-89d94b1d5104", "metadata": {}, "source": [ - "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageable groups such as by day or by station. This allows you to process one group at a time.\n", + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageable groups. This allows you to process one group at a time.\n", "\n", "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", "\n", @@ -454,14 +493,25 @@ "await plan.plan()\n", "print(plan) # preview the plan's request/group counts before any data is fetched\n", "\n", - "\n", "def summarize(table):\n", " # Print a quick summary of one group's table.\n", " d = pl.from_arrow(table)\n", - " stations = d[\"igs\"].unique().sort().to_list()\n", - " if len(stations) > 10:\n", - " stations = f\"{len(stations)} stations\"\n", - " print(len(d), d[\"timestamp\"].min(), d[\"timestamp\"].max(), stations)" + "\n", + " # Mean SNR per station for this group: which sites have better signals?\n", + " snr_by_station = (\n", + " d.group_by(\"igs\")\n", + " .agg(pl.col(\"snr\").mean().round(1).alias(\"mean_snr\"))\n", + " .sort(\"mean_snr\", descending=True)\n", + " )\n", + "\n", + " # Other things you might compute:\n", + " # - Weak signals that may indicate obstruction or hardware trouble\n", + " # d.filter(pl.col(\"snr\") < 25).group_by(\"igs\").len()\n", + " # - Append each group to disk and process the whole week later\n", + " # # d.write_parquet(f\"{OUTPUT_DIR}/{d['timestamp'].min():%Y%m%d}.parquet\")\n", + " # -------------------------------------------------------------------\n", + " print(f\"{d['timestamp'].min()} → {d['timestamp'].max()}\")\n", + " print(snr_by_station)" ] }, { @@ -469,15 +519,7 @@ "id": "6f59fcbd-b77a-4733-82cc-b064b081bc66", "metadata": {}, "source": [ - "Printing a plan reports how many API requests and groups it will run before any data moves. This provides the users with a quick way to gauge how large a query is." - ] - }, - { - "cell_type": "markdown", - "id": "ac24a294-a48d-416a-a285-a2abd1f633c5", - "metadata": {}, - "source": [ - "Processing the plan one day at a time. You never hold more than a single day in memory at once." + "Processing the plan one group at a time. Each group is fetched, processed, and released before the next begins, so the full week never sits in memory at once." ] }, { @@ -515,7 +557,7 @@ "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", "metadata": {}, "source": [ - " > **Check**: Each line of output represents one group." + " > **Check**: Each timestamp range and table represents one group." ] }, { @@ -559,7 +601,7 @@ "\n", "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"2L\"`) with `field=\"snr\"`. How much smaller is the result?\n", "\n", - "3. **Save your results:** Write a fetched dataframe to the scratch directory (*Hint: Use `os.environ[\"SCRATCH_BUCKET\"]`*) as Parquet with `df.write_parquet(...)`, then read it back. Parquet preserves types and is far smaller than CSV." + "3. **Save a different result:** You saved `df_sliced` above. Do the same for the full dataframe `df` from Section 3, giving it a different filename. Compare the two file sizes — how much did filtering the request save you?" ] }, { @@ -582,6 +624,7 @@ "### Further Resources\n", "\n", "* [EarthScope SDK Documentation](https://docs.earthscope.org/sdk)\n", + "* [Authentication using EarthScope CLI](https://gitlab.com/earthscope/public/earthscope-cli)(For non-GeoLab environments)\n", "* [SDK GNSS Observation tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" From e0002f4f5fea440b80d61fec069f481f010b2a5f Mon Sep 17 00:00:00 2001 From: Eshanta Mishra Date: Thu, 6 Aug 2026 17:40:28 +0000 Subject: [PATCH 6/6] Address review comments from Alex --- .../NB1-access-gnss-via-SDK.ipynb | 36 ++++++++++--------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb index 14926f6..0a5d4b3 100644 --- a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb +++ b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb @@ -14,6 +14,7 @@ "metadata": {}, "source": [ "**Version:** 1.0 | **Last updated:** 2026-07-09 \n", + "\n", "**Author:** Eshanta Mishra | **Author institution:** EarthScope Consortium\n", "\n", "**Maintainer:** EarthScope OnRamp Team | **Maintainer's contact :** help@earthscope.org\n", @@ -32,7 +33,7 @@ "\n", "**What this notebook does:** It instantiates the EarthScope SDK and allows you to retrieve GNSS observations for your station of interest.\n", "\n", - "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how modern workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", "\n", "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", "* instantiating the EarthScope SDK client\n", @@ -67,8 +68,8 @@ "\n", "By the end of this notebook, you will be able to:\n", "\n", - "1. Instantiate EarthScope SDK client\n", - "2. Retrieve and filter GNSS data from EarthScope" + "1. Understand the basics of GNSS data available through the EarthScope SDK.\n", + "2. Use the EarthScope SDK client to retreive GNSS data from EarthScope." ] }, { @@ -111,7 +112,7 @@ "id": "9e03f7ee-00b4-4501-bd77-551d49ca52a6", "metadata": {}, "source": [ - "A GNSS (Global Navigation Satellite System) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", + "A Global Navigation Satellite System (GNSS) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", "\n", "Each of those recordings is a GNSS observation. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", "\n", @@ -167,7 +168,7 @@ "id": "52c5e185-8dca-42ff-8a9d-89c23297a761", "metadata": {}, "source": [ - "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are always available inside it. So, the client authenticates automatically. This removes the step of logging in or passing tokens manually, as you would do in a non GeoLab Environment. However, if you are running this notebook outside geolab, you will need to authenticate manually using your EarthScope account credentials through the EarthScope CLI (Detailed instructions for doing this can be found [here](https://gitlab.com/earthscope/public/earthscope-cli))." + "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are already available inside it. The client finds your credentials automatically. This removes the step of logging in again using the CLI or passing tokens manually, as you would do in a non-GeoLab Environment. If you are running this notebook outside geolab, you will need to authenticate using the EarthScope CLI (detailed instructions for doing this can be found [here](https://gitlab.com/earthscope/public/earthscope-cli))." ] }, { @@ -183,7 +184,7 @@ "id": "421226b8-42bc-4d91-a517-469c29fa6e85", "metadata": {}, "source": [ - "`AsyncEarthScopeClient` is the asynchronous client. [Async](https://docs.earthscope.org/sdk/usage#async-usage) lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer." + "`AsyncEarthScopeClient` is the asynchronous client. [Async](https://docs.earthscope.org/sdk/usage#async-usage) lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer. The methods used below are identical on the synchronous client, just remove the `async` / `await` keywords. See [Sync-vs-Async programming](https://www.geeksforgeeks.org/javascript/synchronous-and-asynchronous-programming/) for more information on sync and async programming principles." ] }, { @@ -215,7 +216,7 @@ "source": [ "# Modify these values before running the notebook.\n", "\n", - "STATION = \"AC60\" # GNSS station (4-character ID)\n", + "STATION = \"AC6000USA\" # GNSS station (9-character ID)\n", "SESSION = \"A\" # Session name\n", "START = dt.datetime(2025, 7, 20, 21) # Query start (UTC)\n", "END = dt.datetime(2025, 7, 21, 3) # Query end (UTC)\n", @@ -240,7 +241,7 @@ "source": [ "In this step, we will retrieve GNSS observations for a single station and time window as an Apache Arrow table through the [EarthScope API](https://api.earthscope.org/beta/docs#get-/data-products/gnss/observations).\n", "\n", - "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", + "[Arrow](https://arrow.apache.org/) is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", "\n", "The expected output is a single table with one row per satellite, per signal, per epoch" ] @@ -268,7 +269,7 @@ "id": "b5d3fab2-0ab0-4231-9681-b6e56a117f9d", "metadata": {}, "source": [ - "Convert the Arrow table to a Polars dataframe with `pl.from_arrow(...)`. This is zero-copy, so it is very efficient. Sorting by `timestamp` makes the rows read chronologically." + "Convert the Arrow table to a [Polars](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html) dataframe with `pl.from_arrow(...)`. This is zero-copy, so it is very efficient. Sorting by `timestamp` makes the rows read chronologically." ] }, { @@ -296,7 +297,7 @@ "metadata": {}, "source": [ "The default data query returns every field - the same information you would find in a RINEX observation file. Let us take a moment to understand the data structure.\n", - "Each row of data is a single measurement (one satellite, one signal, one instant). Inspect the columns, the constellations present, and the range of signal strengths." + "Each row is a single measurement (one satellite, one signal, one instant). Inspect the columns, the constellations present, and the range of signal strengths." ] }, { @@ -325,8 +326,8 @@ "| `satellite` | int | Satellite number within its constellation (its PRN / slot). Combine with `system` for a globally unique ID. |\n", "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L` (decoded below). |\n", "| `range` | float | Pseudorange in meters. It is the apparent satellite to receiver distance. |\n", - "| `phase` | float | Carrier phase in cycles. It is a precise but ambiguous range measurement. |\n", - "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher is cleaner. |\n", + "| `phase` | float | Carrier phase in cycles. It is used to drive a precise but ambiguous range measurement. |\n", + "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher values correspond to stronger signal. |\n", "| `slip` | int | Cycle-slip indicator; `null` when phase tracking was continuous. |\n", "| `flags` | int | Per-observation status flags (e.g. loss-of-lock). |\n", "| `fcn` | int | Frequency channel number — used by GLONASS's FDMA signals; `0` otherwise. |\n", @@ -368,9 +369,9 @@ "id": "c0a4e066-7b6b-4161-9260-a4a825637f2a", "metadata": {}, "source": [ - "When you are working with RINEX files, the files hand you everything, all fields, constellations, satellites etc. Most analyses only need a small slice of these data. Passing filters to `gnss_observations()` in the SDK pushes that selection to the server, so only the data that you asked for is transferred.\n", + "When you are working with RINEX files, the files include everything: all fields, constellations, satellites etc. Many analyses only need a small slice of these data. Passing filters to `gnss_observations()` in the SDK pushes that selection to the server, so only the data that you asked for is transferred.\n", "\n", - "The payoff is immediate: requesting a single signal from a single satellite over two months returns in a fraction of the time and requires smaller memory size, where full RINEX for the same window would be orders of magnitude larger and mostly discarded. You also skip RINEX parsing entirely because the results arrive as an Arrow table ready to load straight into a dataframe, rather than a set of daily text files you have to decode first.\n", + "The payoff is immediate: requesting a single signal from a single satellite over two months returns in a fraction of the time and requires less memory, where full RINEX for the same window would be orders of magnitude larger and mostly discarded. You also skip RINEX parsing entirely because the results arrive as an Arrow table, rather than a set of daily text files you have to decode first.\n", "\n", "*A smaller slice also means less data moved across the network, which keeps costs down on EarthScope's side.*\n", "\n", @@ -423,7 +424,7 @@ "| Location | Path | Persistence | Use it for |\n", "|---|---|---|---|\n", "| **Home** | `/home/jovyan/` | Private, persistent, 50 GB limit | Notebooks, scripts, and results you want to keep while working on your project |\n", - "| **Shared** | `/home/jovyan/shared/` | Read-only | Datasets and starter notebooks placed there by instructors. You can copy files out, but not modify them |\n", + "| **Shared** | `/home/jovyan/shared/` | Read-only | Datasets and starter notebooks placed there by instructors. You can copy files out, but not modify the original |\n", "| **Scratch** | `/earthscope-scratch/` (exact path in `os.environ[\"SCRATCH_BUCKET\"]`) | Temporary and auto-deletes after ~2 weeks | Large intermediates over the 50 GB home limit |\n", "\n", "Use Parquet rather than CSV. It's columnar, compressed, and preserves data types — your `timestamp` comes back as a datetime and `snr` as a float, with no re-parsing." @@ -597,7 +598,7 @@ "\n", "**Try these modifications:**\n", "\n", - "1. **Change the station:** Set `STATION` in the Configuration section to a different 4-character station ID and re-run Section 3. Does the station report the same constellations?\n", + "1. **Change the station:** Set `STATION` in the Configuration section to a different 9-character station ID and re-run Section 3. Does the station report the same constellations?\n", "\n", "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"2L\"`) with `field=\"snr\"`. How much smaller is the result?\n", "\n", @@ -627,7 +628,8 @@ "* [Authentication using EarthScope CLI](https://gitlab.com/earthscope/public/earthscope-cli)(For non-GeoLab environments)\n", "* [SDK GNSS Observation tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", - "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" + "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)\n", + "* [Sync-vs-Async programming](https://www.geeksforgeeks.org/javascript/synchronous-and-asynchronous-programming/)" ] } ],