diff --git a/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb new file mode 100644 index 0000000..1d9c2a6 --- /dev/null +++ b/tutorials/mvp-1-geodesy/.ipynb_checkpoints/NB1-access-gnss-via-SDK-checkpoint.ipynb @@ -0,0 +1,614 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2abe5d1d-4a2f-4c96-a78c-6c3e5f9ac750", + "metadata": {}, + "source": [ + "# Accessing GNSS Observations with the EarthScope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "35320cb4-aed2-4420-8f9c-b3deb365e792", + "metadata": {}, + "source": [ + "**Version:** 1.0 | **Last updated:** 2026-07-09 | **Author:** Eshanta Mishra" + ] + }, + { + "cell_type": "markdown", + "id": "b9b1d1c3-569f-459a-8595-e1ba758e4bc7", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What this notebook does:** It instantiates the earthscope SDK and allows you to retrieve GNSS observations for your station of interest.\n", + "\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "\n", + "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", + "* instantiating the EarthScope SDK client\n", + "* Select data for stations using the station name and time range.\n", + "* Retrieve observations and load into dataframes.\n", + "* Filter (slice) the GNSS data based on different parameters.\n", + "\n", + "---\n", + "\n", + "### Prerequisites\n", + "\n", + "\n", + "Before starting this notebook, you should:\n", + "* [ ] This is an introductory notebook. Being familiar with **Geodesy, GNSS, python dataframes** is recommended but not required.\n", + "\n", + "---\n", + "\n", + "### GeoLab Compute Resources\n", + "\n", + "| Setting | Recommended |\n", + "|---|---|\n", + "| **Image** | GeoLab (default image)|\n", + "| **Server size** | 4 GB RAM, ~0.5 CPUs (default server) |" + ] + }, + { + "cell_type": "markdown", + "id": "811fc68e-1e81-482c-82dd-5efa2a89d715", + "metadata": {}, + "source": [ + "## Learning Objectives\n", + "\n", + "By the end of this notebook, you will be able to:\n", + "\n", + "1. Instantiate EarthScope SDK client\n", + "2. Retrieve and filter GNSS data from EarthScope" + ] + }, + { + "cell_type": "markdown", + "id": "1ec3d79a-ad70-4c58-b8fc-287501adb7e9", + "metadata": {}, + "source": [ + "## Relevant Documentation & Resources\n", + "\n", + "* [EarthScope SDK documentation](https://docs.earthscope.org/sdk)" + ] + }, + { + "cell_type": "markdown", + "id": "076d400d-8c88-4113-a28d-91dac8648679", + "metadata": {}, + "source": [ + "## Contents\n", + "\n", + "1. [Basics of GNSS](#id-1-basics-of-gnss)\n", + "2. [Setup & Imports](#id-2-setup-imports)\n", + "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", + "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", + "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", + "6. [Larger than Memory Requests with Query Plans](#id-6-larger-than-memory-requests-with-query-plans)\n", + "7. [Exploration Exercises](#id-7-exploration-exercises)\n", + "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" + ] + }, + { + "cell_type": "markdown", + "id": "1b245a5d-5792-461d-bd51-99e55ab797a0", + "metadata": {}, + "source": [ + "## 1. Basics of GNSS" + ] + }, + { + "cell_type": "markdown", + "id": "9e03f7ee-00b4-4501-bd77-551d49ca52a6", + "metadata": {}, + "source": [ + "A GNSS (Global Navigation Satellite System) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", + "\n", + "Each of those recordings is a **GNSS observation**. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", + "\n", + "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", + "2. **Carrier phase**: It is a more precise distance measurement between the satellite and the ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", + "3. **SNR**: Signal-to-Noise Ratio. It is a parameter that tells how strong and clean the received signal is.\n", + "\n", + "**Raw Observations vs. derived products**\n", + "\n", + "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " + ] + }, + { + "cell_type": "markdown", + "id": "cca79b91-0348-4ee2-842e-740595288820", + "metadata": {}, + "source": [ + "## 2. Setup & Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8cd2903c-1b18-4877-a421-f19374f8fcf3", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime as dt\n", + "import polars as pl\n", + "from earthscope_sdk import AsyncEarthScopeClient" + ] + }, + { + "cell_type": "markdown", + "id": "50263265-10c7-44c7-a302-37b658b899c8", + "metadata": {}, + "source": [ + "### Instantiating the EarthScope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "8683b5f9-3ffc-4eb4-a8ec-82c61fa196e7", + "metadata": {}, + "source": [ + "#### Authentication" + ] + }, + { + "cell_type": "markdown", + "id": "52c5e185-8dca-42ff-8a9d-89c23297a761", + "metadata": {}, + "source": [ + "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are always available inside it. So, the client authenticates automatically. This removes the step of logging in or passing tokens manually, as you would do in a non GeoLab Environment." + ] + }, + { + "cell_type": "markdown", + "id": "b0143b85-f435-4ce2-9359-6b5f2b8d3674", + "metadata": {}, + "source": [ + "#### Async client" + ] + }, + { + "cell_type": "markdown", + "id": "421226b8-42bc-4d91-a517-469c29fa6e85", + "metadata": {}, + "source": [ + "`AsyncEarthScopeClient` is the asynchronous client. Async lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "abe833e0-9ca9-48bd-8bfc-749579592f92", + "metadata": {}, + "outputs": [], + "source": [ + "es = AsyncEarthScopeClient()" + ] + }, + { + "cell_type": "markdown", + "id": "aa3e8c77-ceaf-4f6e-be2e-b99b0b7f0125", + "metadata": {}, + "source": [ + "### Configuration\n", + "\n", + "Set your parameters here before running the rest of the notebook. Every subsequent cells read from these variables. So, this is the only place you need to edit to point the notebook at different data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15ecb388-d718-4949-a072-5db6cd398d65", + "metadata": {}, + "outputs": [], + "source": [ + "# Modify these values before running the notebook.\n", + "\n", + "STATION = \"AC60\" # GNSS station (4-character ID)\n", + "SESSION = \"A\" # Session name\n", + "START = dt.datetime(2025, 7, 20, 21) # Query start (UTC)\n", + "END = dt.datetime(2025, 7, 21, 3) # Query end (UTC)" + ] + }, + { + "cell_type": "markdown", + "id": "f2e9e575-a7b1-4dd8-86a9-17f0fd942f15", + "metadata": {}, + "source": [ + "## 3. Retrieve Observations for a single station" + ] + }, + { + "cell_type": "markdown", + "id": "af77148f-64a0-424a-9dc2-2ec34a8f96e8", + "metadata": {}, + "source": [ + "In this step, we will retrieve GNSS observations for a single station and time window as an Apache arrow table through the EarthScope data API.\n", + "\n", + "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", + "\n", + "The Expected output is a single table with one row per satellite, per signal, per epoch" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4eb07e18-b72b-4ed4-99b7-558d935fee31", + "metadata": {}, + "outputs": [], + "source": [ + "# Step: describe the request, then .fetch() to run the query.\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=START,\n", + " end_datetime=END,\n", + " station_name=STATION,\n", + " session_name=SESSION,\n", + ").fetch()\n", + "\n", + "table" + ] + }, + { + "cell_type": "markdown", + "id": "b5d3fab2-0ab0-4231-9681-b6e56a117f9d", + "metadata": {}, + "source": [ + "Convert the Arrow table to a Polars dataframe with `pl.from_arrow(...)`. This is zero-copy, so it is very efficient. Sorting by `timestamp` makes the rows read chronologically." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "82b54655-559f-4b95-8479-a386affe5b91", + "metadata": {}, + "outputs": [], + "source": [ + "df = pl.from_arrow(table).sort(\"timestamp\")\n", + "df" + ] + }, + { + "cell_type": "markdown", + "id": "05678a5b-388b-4d44-8e04-28a8912af92e", + "metadata": {}, + "source": [ + "## 4. Understanding the Observation fields" + ] + }, + { + "cell_type": "markdown", + "id": "6f775aef-3b48-4a78-9d4f-b43b7cd1c143", + "metadata": {}, + "source": [ + "The default data query returns every field - the same information you would find in a RINEX observation file. Let us take a moment to understand the data structure.\n", + "Each row of data is a single measurement (one satellite, one signal, one instant). Inspect the columns, the constellations present, and the range of signal strengths." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1ab79788-7803-4711-aa4f-f977b8e83e7a", + "metadata": {}, + "outputs": [], + "source": [ + "# Inspect structure and coverage\n", + "print(df.schema) # column names and types\n", + "print(\"Constellations:\", df[\"system\"].unique().to_list())\n", + "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())\n", + "\n", + "df.describe() # summary statistics" + ] + }, + { + "cell_type": "markdown", + "id": "c59fb35b-ec60-4879-8a0e-1c6bbe9e4a4a", + "metadata": {}, + "source": [ + "### What each field means\n", + "\n", + "| Column | Type | Meaning |\n", + "|---|---|---|\n", + "| `timestamp` | datetime (UTC) | The epoch of the observation. |\n", + "| `satellite` | int | Satellite number within its constellation (its PRN / slot). Combine with `system` for a globally unique ID. |\n", + "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L`, `5Q` (decoded below). |\n", + "| `range` | float | Pseudorange in meters. It is the apparent satellite to receiver distance. |\n", + "| `phase` | float | Carrier phase in cycles. It is a precise but ambiguous range measurement. |\n", + "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher is cleaner. |\n", + "| `slip` | int | Cycle-slip indicator; `null` when phase tracking was continuous. |\n", + "| `flags` | int | Per-observation status flags (e.g. loss-of-lock). |\n", + "| `fcn` | int | Frequency channel number — used by GLONASS's FDMA signals; `0` otherwise. |\n", + "| `system` | str | Constellation code (see below). |\n", + "| `igs` | str | The station's IGS long name (e.g. `AC6000USA`). |\n", + "\n", + "### Reading the `system` column\n", + "\n", + "| Code | Constellation |\n", + "|---|---|\n", + "| `G` | GPS (USA) |\n", + "| `R` | GLONASS (Russia) |\n", + "| `E` | Galileo (EU) |\n", + "| `C` | BeiDou (China) |\n", + "| `J` | QZSS (Japan) |\n", + "| `I` | NavIC / IRNSS (India) |\n", + "| `S` | SBAS (augmentation) |\n", + "\n", + "### Reading the `obs_code` column\n", + "\n", + "An observation code names a specific signal as **band + tracking attribute**:\n", + "\n", + "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", + "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", + "\n", + "For example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." + ] + }, + { + "cell_type": "markdown", + "id": "7f1cd227-e581-4790-97a7-176bfc846f2a", + "metadata": {}, + "source": [ + "## 5. Requesting Only the Data You Need" + ] + }, + { + "cell_type": "markdown", + "id": "c0a4e066-7b6b-4161-9260-a4a825637f2a", + "metadata": {}, + "source": [ + "When you are working with RINEX files, the files hand you everything, all fields, constellations, satellites etc. Most analyses only need a small slice of these data. Passing filters to `gnss_observations()` in the SDK pushes that selection to the server, so only the data that you asked for is transferred.\n", + "\n", + "Downloading a single signal from a single satellite over two months is drastically cheaper than downloading full RINEX and discarding 99% of it. Less data means faster fetches and smaller memory footprints.\n", + "\n", + "The resulting output is a dataframe containing only the requested stations, constellations, satellites, signals and fields. For example, only `snr` and `range` columns when `field=[\"snr\",\"range\"]`.\n", + "\n", + "Each filter below is optional, accepts a single value or a list, and shrinks the request:\n", + "\n", + "* `station_name`: one name or a list of names\n", + "* `network_name`: request a whole network at once\n", + "* `system`: constellation code(s), e.g. `\"G\"` or `[\"G\", \"R\"]`\n", + "* `satellite`: specific satellite number(s)\n", + "* `obs_code`: specific signal(s), e.g. `[\"1C\", \"2L\"]`\n", + "* `field`: which measurement column(s) to return, e.g. `[\"snr\", \"range\"]`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "949a6db5-f949-43f9-930c-88344765e60b", + "metadata": {}, + "outputs": [], + "source": [ + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " station_name=[\"P717\", \"P453\", \"P146\", \"P147\", \"P041\"],\n", + " session_name=\"A\",\n", + " system=[\"G\", \"R\"],\n", + " obs_code=[\"1C\", \"2L\"],\n", + " satellite=[\"7\", \"21\", \"28\"],\n", + " field=[\"snr\", \"range\"],\n", + ").fetch()\n", + "\n", + "df_sliced = pl.from_arrow(table).sort(\"timestamp\")\n", + "df_sliced" + ] + }, + { + "cell_type": "markdown", + "id": "0cad076e-a951-416e-ad88-96dd7fe1f253", + "metadata": {}, + "source": [ + "> **Check:** The result should contain only the `snr` and `range` measurement columns, no `phase`, no `flags`." + ] + }, + { + "cell_type": "markdown", + "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", + "metadata": {}, + "source": [ + "## 6. Larger than Memory Requests with Query Plans" + ] + }, + { + "cell_type": "markdown", + "id": "81e989fa-baf0-4396-920e-fece9f620f43", + "metadata": {}, + "source": [ + "### What is a query plan?" + ] + }, + { + "cell_type": "markdown", + "id": "7a61354a-548f-4459-976c-89d94b1d5104", + "metadata": {}, + "source": [ + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageable groups such as by day or by station. This allows you to process one group at a time.\n", + "\n", + "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", + "\n", + "The expected output is a summary for each group (row count, time span, and stations), rather than one huge dataframe." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1772dd83-792b-4666-a107-392d8522b92f", + "metadata": {}, + "outputs": [], + "source": [ + "# Build a query plan (note: no .fetch())\n", + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "plan = es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " network_name=\"PERM:Alaska\",\n", + " session_name=\"A\",\n", + " system=\"G\",\n", + " field=[\"phase\", \"range\", \"snr\"],\n", + ")\n", + "\n", + "print(plan) # preview the plan's request/group counts before any data is fetched\n", + "\n", + "\n", + "def summarize(table):\n", + " # Print a quick summary of one group's table.\n", + " d = pl.from_arrow(table)\n", + " stations = d[\"igs\"].unique().sort().to_list()\n", + " if len(stations) > 10:\n", + " stations = f\"{len(stations)} stations\"\n", + " print(len(d), d[\"timestamp\"].min(), d[\"timestamp\"].max(), stations)" + ] + }, + { + "cell_type": "markdown", + "id": "6f59fcbd-b77a-4733-82cc-b064b081bc66", + "metadata": {}, + "source": [ + "Printing a plan reports how many API requests and groups it will run before any data moves. This provides the users with a quick way to gauge how large a query is." + ] + }, + { + "cell_type": "markdown", + "id": "ac24a294-a48d-416a-a285-a2abd1f633c5", + "metadata": {}, + "source": [ + "Processing the plan one day at a time. You never hold more than a single day in memory at once." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "46d1af54-622f-4066-9895-cfd595341cdd", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_day():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "02c76245-1e7a-4663-b398-f0472a58eb68", + "metadata": {}, + "source": [ + "Processing the plan one station at a time across the whole window instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "34742255-bd16-4197-b151-1777b4753a1d", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_station():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", + "metadata": {}, + "source": [ + " > **Check**: Each line of output represents one group." + ] + }, + { + "cell_type": "markdown", + "id": "8e88b724-65ec-4ef7-9e19-fb6bbf8c895b", + "metadata": {}, + "source": [ + "### A rough guide for choosing a strategy" + ] + }, + { + "cell_type": "markdown", + "id": "d0673a80-e5e9-4256-8f3f-a48ba2ddafa2", + "metadata": {}, + "source": [ + "* Less than a week: `.fetch()` the whole results at once.\n", + "* Weeks to months, or many stations: iterate with `group_by_day()` (or `group_by_station()`)\n", + "* Months to years: define custom batches with `plan.group_by()`." + ] + }, + { + "cell_type": "markdown", + "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", + "metadata": {}, + "source": [ + "For custom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [EarthScope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." + ] + }, + { + "cell_type": "markdown", + "id": "c5008800-7811-4fbc-9721-9869da927fc2", + "metadata": {}, + "source": [ + "## 7. Exploration Exercises\n", + "\n", + "Now that you've completed the core workflow, try modifying the parameters below to explore how the results change.\n", + "\n", + "**Try these modifications:**\n", + "\n", + "1. **Change the station:** Set `STATION` in the Configuration section to a different 4-character station ID and re-run Section 3. Does the station report the same constellations?\n", + "\n", + "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"5Q\"`) with `field=\"snr\"`. How much smaller is the result?\n", + "\n", + "3. **Save your results:** Write a fetched dataframe to the scratch directory (*Hint: Use `os.environ[\"SCRATCH_BUCKET\"]`*) as Parquet with `df.write_parquet(...)`, then read it back. Parquet preserves types and is far smaller than CSV." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "edf9b94a-b045-4fcf-be34-f2e2f958f124", + "metadata": {}, + "outputs": [], + "source": [ + "# Exploration cell — use this space to experiment" + ] + }, + { + "cell_type": "markdown", + "id": "1e147f5a-2352-4a16-b28b-3f5c1dbea902", + "metadata": {}, + "source": [ + "## 8. Troubleshooting & Support\n", + "\n", + "### Further Resources\n", + "\n", + "* [EarthScope SDK Documentation](https://docs.earthscope.org/sdk)\n", + "* [SDK GNSS Observation tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", + "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", + "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb new file mode 100644 index 0000000..a69fe75 --- /dev/null +++ b/tutorials/mvp-1-geodesy/NB1-access-gnss-via-SDK.ipynb @@ -0,0 +1,612 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2abe5d1d-4a2f-4c96-a78c-6c3e5f9ac750", + "metadata": {}, + "source": [ + "# Accessing GNSS Observations with the EarthScope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "35320cb4-aed2-4420-8f9c-b3deb365e792", + "metadata": {}, + "source": [ + "**Version:** 1.0 | **Last updated:** 2026-07-09 | **Author:** Eshanta Mishra" + ] + }, + { + "cell_type": "markdown", + "id": "b9b1d1c3-569f-459a-8595-e1ba758e4bc7", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What this notebook does:** It instantiates the EarthScope SDK and allows you to retrieve GNSS observations for your station of interest.\n", + "\n", + "**Why it is useful:** This notebook provides a hands-on on how to use the SDK to retrieve GNSS observations from the cloud and load it into a dataframe. It provides the users an insight into how cloud based workflows can be more efficient compared to traditional methods of downloading and accessing GNSS observations as RINEX files.\n", + "\n", + "**What you will accomplish:** By the end of this notebook, we will have accomplished:\n", + "* instantiating the EarthScope SDK client\n", + "* Select data for stations using the station name and time range.\n", + "* Retrieve observations and load into dataframes.\n", + "* Filter (slice) the GNSS data based on different parameters.\n", + "\n", + "---\n", + "\n", + "### Prerequisites\n", + "\n", + "\n", + "Before starting this notebook, you should:\n", + "* [ ] This is an introductory notebook. Being familiar with **Geodesy, GNSS, python dataframes** is recommended but not required.\n", + "\n", + "---\n", + "\n", + "### GeoLab Compute Resources\n", + "\n", + "| Setting | Recommended |\n", + "|---|---|\n", + "| **Image** | GeoLab (default image)|\n", + "| **Server size** | 4 GB RAM, ~0.5 CPUs (default server) |" + ] + }, + { + "cell_type": "markdown", + "id": "811fc68e-1e81-482c-82dd-5efa2a89d715", + "metadata": {}, + "source": [ + "## Learning Objectives\n", + "\n", + "By the end of this notebook, you will be able to:\n", + "\n", + "1. Instantiate EarthScope SDK client\n", + "2. Retrieve and filter GNSS data from EarthScope" + ] + }, + { + "cell_type": "markdown", + "id": "1ec3d79a-ad70-4c58-b8fc-287501adb7e9", + "metadata": {}, + "source": [ + "## Relevant Documentation & Resources\n", + "\n", + "* [EarthScope SDK documentation](https://docs.earthscope.org/sdk)" + ] + }, + { + "cell_type": "markdown", + "id": "076d400d-8c88-4113-a28d-91dac8648679", + "metadata": {}, + "source": [ + "## Contents\n", + "\n", + "1. [Basics of GNSS](#id-1-basics-of-gnss)\n", + "2. [Setup & Imports](#id-2-setup-imports)\n", + "3. [Retrieve Observations for a Single Station](#id-3-retrieve-observations-for-a-single-station)\n", + "4. [Understanding the Observation fields](#id-4-understanding-the-observation-fields)\n", + "5. [Requesting Only the Data you Need](#id-5-requesting-only-the-data-you-need)\n", + "6. [Larger than Memory Requests with Query Plans](#id-6-larger-than-memory-requests-with-query-plans)\n", + "7. [Exploration Exercises](#id-7-exploration-exercises)\n", + "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" + ] + }, + { + "cell_type": "markdown", + "id": "1b245a5d-5792-461d-bd51-99e55ab797a0", + "metadata": {}, + "source": [ + "## 1. Basics of GNSS" + ] + }, + { + "cell_type": "markdown", + "id": "9e03f7ee-00b4-4501-bd77-551d49ca52a6", + "metadata": {}, + "source": [ + "A GNSS (Global Navigation Satellite System) is a constellation of satellites that broadcast timing signals. GNSS includes constellations such as GPS (United States), GLONASS (Russia), BeiDou (China) etc. A ground station (receiver plus antenna) records the data transmitted by these satellites several times per minute.\n", + "\n", + "Each of those recordings is a GNSS observation. It is a raw measurement of the signal at a given instant for a given satellite, on one frequency. Some commonly used observations are:\n", + "\n", + "1. **Pseudorange**: Apparent distance to the satellite, in meters. It is called *pseudo* because the receiver and satellite clocks aren't perfectly synchronized.\n", + "2. **Carrier phase**: It is a more precise distance measurement between the satellite and the ground, counted in whole cycles of the carrier wave (with an unknown starting offset)\n", + "3. **SNR**: Signal-to-Noise Ratio. It is a parameter that tells how strong and clean the received signal is.\n", + "\n", + "**Raw Observations vs. derived products**\n", + "\n", + "The data we retrieve in this notebook are raw observations. The geodetic results you may ultimately want such as a station's position, displacement over time, etc. are derived products. These derived products are computed by processing many observations together. " + ] + }, + { + "cell_type": "markdown", + "id": "cca79b91-0348-4ee2-842e-740595288820", + "metadata": {}, + "source": [ + "## 2. Setup & Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8cd2903c-1b18-4877-a421-f19374f8fcf3", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime as dt\n", + "import polars as pl\n", + "from earthscope_sdk import AsyncEarthScopeClient" + ] + }, + { + "cell_type": "markdown", + "id": "50263265-10c7-44c7-a302-37b658b899c8", + "metadata": {}, + "source": [ + "### Instantiating the EarthScope SDK" + ] + }, + { + "cell_type": "markdown", + "id": "8683b5f9-3ffc-4eb4-a8ec-82c61fa196e7", + "metadata": {}, + "source": [ + "#### Authentication" + ] + }, + { + "cell_type": "markdown", + "id": "52c5e185-8dca-42ff-8a9d-89c23297a761", + "metadata": {}, + "source": [ + "Since you use your EarthScope account to log into GeoLab, your EarthScope credentials are always available inside it. So, the client authenticates automatically. This removes the step of logging in or passing tokens manually, as you would do in a non GeoLab Environment." + ] + }, + { + "cell_type": "markdown", + "id": "b0143b85-f435-4ce2-9359-6b5f2b8d3674", + "metadata": {}, + "source": [ + "#### Async client" + ] + }, + { + "cell_type": "markdown", + "id": "421226b8-42bc-4d91-a517-469c29fa6e85", + "metadata": {}, + "source": [ + "`AsyncEarthScopeClient` is the asynchronous client. Async lets the SDK process a large query into many concurrent sub-requests, making large quantity of data retrieval more efficient. The SDK also comes with a synchronous `EarthScopeClient` if you prefer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "abe833e0-9ca9-48bd-8bfc-749579592f92", + "metadata": {}, + "outputs": [], + "source": [ + "es = AsyncEarthScopeClient()" + ] + }, + { + "cell_type": "markdown", + "id": "aa3e8c77-ceaf-4f6e-be2e-b99b0b7f0125", + "metadata": {}, + "source": [ + "### Configuration\n", + "\n", + "Set your parameters here before running the rest of the notebook. Every subsequent cells read from these variables. So, this is the only place you need to edit to point the notebook at different data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15ecb388-d718-4949-a072-5db6cd398d65", + "metadata": {}, + "outputs": [], + "source": [ + "# Modify these values before running the notebook.\n", + "\n", + "STATION = \"AC60\" # GNSS station (4-character ID)\n", + "SESSION = \"A\" # Session name\n", + "START = dt.datetime(2025, 7, 20, 21) # Query start (UTC)\n", + "END = dt.datetime(2025, 7, 21, 3) # Query end (UTC)" + ] + }, + { + "cell_type": "markdown", + "id": "f2e9e575-a7b1-4dd8-86a9-17f0fd942f15", + "metadata": {}, + "source": [ + "## 3. Retrieve Observations for a single station" + ] + }, + { + "cell_type": "markdown", + "id": "af77148f-64a0-424a-9dc2-2ec34a8f96e8", + "metadata": {}, + "source": [ + "In this step, we will retrieve GNSS observations for a single station and time window as an Apache Arrow table through the EarthScope data API.\n", + "\n", + "Arrow is a fast, columnar, in-memory format that most dataframe libraries read with little or no copying. The API also lets you request an arbitrary time window, unlike the daily data chunks of RINEX files (RINEX provides one file per station per UTC day).\n", + "\n", + "The expected output is a single table with one row per satellite, per signal, per epoch" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4eb07e18-b72b-4ed4-99b7-558d935fee31", + "metadata": {}, + "outputs": [], + "source": [ + "#describe the request, then .fetch() to run the query.\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=START,\n", + " end_datetime=END,\n", + " station_name=STATION,\n", + " session_name=SESSION,\n", + ").fetch()\n", + "\n", + "table" + ] + }, + { + "cell_type": "markdown", + "id": "b5d3fab2-0ab0-4231-9681-b6e56a117f9d", + "metadata": {}, + "source": [ + "Convert the Arrow table to a Polars dataframe with `pl.from_arrow(...)`. This is zero-copy, so it is very efficient. Sorting by `timestamp` makes the rows read chronologically." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "82b54655-559f-4b95-8479-a386affe5b91", + "metadata": {}, + "outputs": [], + "source": [ + "df = pl.from_arrow(table).sort(\"timestamp\")\n", + "df" + ] + }, + { + "cell_type": "markdown", + "id": "05678a5b-388b-4d44-8e04-28a8912af92e", + "metadata": {}, + "source": [ + "## 4. Understanding the Observation fields" + ] + }, + { + "cell_type": "markdown", + "id": "6f775aef-3b48-4a78-9d4f-b43b7cd1c143", + "metadata": {}, + "source": [ + "The default data query returns every field - the same information you would find in a RINEX observation file. Let us take a moment to understand the data structure.\n", + "Each row of data is a single measurement (one satellite, one signal, one instant). Inspect the columns, the constellations present, and the range of signal strengths." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1ab79788-7803-4711-aa4f-f977b8e83e7a", + "metadata": {}, + "outputs": [], + "source": [ + "# Inspect structure and coverage\n", + "print(df.schema) \n", + "print(\"Constellations:\", df[\"system\"].unique().sort().to_list())\n", + "print(\"Observation codes:\", df[\"obs_code\"].unique().sort().to_list())" + ] + }, + { + "cell_type": "markdown", + "id": "c59fb35b-ec60-4879-8a0e-1c6bbe9e4a4a", + "metadata": {}, + "source": [ + "### What each field means\n", + "\n", + "| Column | Type | Meaning |\n", + "|---|---|---|\n", + "| `timestamp` | datetime (UTC) | The epoch of the observation. |\n", + "| `satellite` | int | Satellite number within its constellation (its PRN / slot). Combine with `system` for a globally unique ID. |\n", + "| `obs_code` | str | Which signal was measured, e.g. `1C`, `2W`, `2L` (decoded below). |\n", + "| `range` | float | Pseudorange in meters. It is the apparent satellite to receiver distance. |\n", + "| `phase` | float | Carrier phase in cycles. It is a precise but ambiguous range measurement. |\n", + "| `snr` | float | Signal strength as carrier-to-noise density ($C/N_0$), roughly in dB-Hz. Higher is cleaner. |\n", + "| `slip` | int | Cycle-slip indicator; `null` when phase tracking was continuous. |\n", + "| `flags` | int | Per-observation status flags (e.g. loss-of-lock). |\n", + "| `fcn` | int | Frequency channel number — used by GLONASS's FDMA signals; `0` otherwise. |\n", + "| `system` | str | Constellation code (see below). |\n", + "| `igs` | str | The station's IGS long name (e.g. `AC6000USA`). |\n", + "\n", + "### Reading the `system` column\n", + "\n", + "| Code | Constellation |\n", + "|---|---|\n", + "| `G` | GPS (USA) |\n", + "| `R` | GLONASS (Russia) |\n", + "| `E` | Galileo (EU) |\n", + "| `C` | BeiDou (China) |\n", + "| `J` | QZSS (Japan) |\n", + "| `I` | NavIC / IRNSS (India) |\n", + "| `S` | SBAS (augmentation) |\n", + "\n", + "### Reading the `obs_code` column\n", + "\n", + "An observation code names a specific signal as **band + tracking attribute**:\n", + "\n", + "* The **digit** is the frequency band: `1` (L1), `2` (L2), `5` (L5), and so on.\n", + "* The **letter** is the tracking mode or signal component: `C` (C/A or civil code), `W` (semi-codeless Z-tracking of the encrypted P-code), and `L` / `Q` / `X` / `I` (specific modern civil components).\n", + "\n", + "For example, `1C` is the classic GPS L1 C/A signal, `2W` is L2 P(Y) via Z-tracking, `2L` is the L2C signal." + ] + }, + { + "cell_type": "markdown", + "id": "7f1cd227-e581-4790-97a7-176bfc846f2a", + "metadata": {}, + "source": [ + "## 5. Requesting Only the Data You Need" + ] + }, + { + "cell_type": "markdown", + "id": "c0a4e066-7b6b-4161-9260-a4a825637f2a", + "metadata": {}, + "source": [ + "When you are working with RINEX files, the files hand you everything, all fields, constellations, satellites etc. Most analyses only need a small slice of these data. Passing filters to `gnss_observations()` in the SDK pushes that selection to the server, so only the data that you asked for is transferred.\n", + "\n", + "Downloading a single signal from a single satellite over two months is drastically cheaper than downloading full RINEX and discarding 99% of it. Less data means faster fetches and smaller memory footprints.\n", + "\n", + "The resulting output is a dataframe containing only the requested stations, constellations, satellites, signals and fields. For example, only `snr` and `range` columns when `field=[\"snr\",\"range\"]`.\n", + "\n", + "Each filter below is optional, accepts a single value or a list, and shrinks the request:\n", + "\n", + "* `station_name`: one name or a list of names\n", + "* `network_name`: request a whole network at once\n", + "* `system`: constellation code(s), e.g. `\"G\"` or `[\"G\", \"R\"]`\n", + "* `satellite`: specific satellite number(s)\n", + "* `obs_code`: specific signal(s), e.g. `[\"1C\", \"2L\"]`\n", + "* `field`: which measurement column(s) to return, e.g. `[\"snr\", \"range\"]`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "949a6db5-f949-43f9-930c-88344765e60b", + "metadata": {}, + "outputs": [], + "source": [ + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "table = await es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " station_name=[\"P717\", \"P453\", \"P146\", \"P147\", \"P041\"],\n", + " session_name=\"A\",\n", + " system=[\"G\", \"R\"],\n", + " obs_code=[\"1C\", \"2L\"],\n", + " satellite=[\"7\", \"21\", \"28\"],\n", + " field=[\"snr\", \"range\"],\n", + ").fetch()\n", + "\n", + "df_sliced = pl.from_arrow(table).sort(\"timestamp\")\n", + "df_sliced" + ] + }, + { + "cell_type": "markdown", + "id": "0cad076e-a951-416e-ad88-96dd7fe1f253", + "metadata": {}, + "source": [ + "> **Check:** The result should contain only the `snr` and `range` measurement columns, no `phase`, no `flags`." + ] + }, + { + "cell_type": "markdown", + "id": "ac4a4474-22ee-4078-a340-3a4b62e9e6f1", + "metadata": {}, + "source": [ + "## 6. Larger than Memory Requests with Query Plans" + ] + }, + { + "cell_type": "markdown", + "id": "81e989fa-baf0-4396-920e-fece9f620f43", + "metadata": {}, + "source": [ + "### What is a query plan?" + ] + }, + { + "cell_type": "markdown", + "id": "7a61354a-548f-4459-976c-89d94b1d5104", + "metadata": {}, + "source": [ + "**What it does**: Instead of calling `.fetch()` which returns one big table, you build a query plan by leaving `.fetch()` off. A query plan is iterable, i.e. it gives the result set in manageable groups such as by day or by station. This allows you to process one group at a time.\n", + "\n", + "**Why it matters**: Some requests such as an entire network for a week will not fit in memory at once. Query plans also limit how many requests hit the API at the same time. Each group's sub-requests run in parallel and are collected into a single table you can process before moving on.\n", + "\n", + "The expected output is a summary for each group (row count, time span, and stations), rather than one huge dataframe." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1772dd83-792b-4666-a107-392d8522b92f", + "metadata": {}, + "outputs": [], + "source": [ + "# Build a query plan (note: no .fetch())\n", + "start = dt.datetime(2025, 7, 20)\n", + "end = start + dt.timedelta(days=7)\n", + "\n", + "plan = es.data.gnss_observations(\n", + " start_datetime=start,\n", + " end_datetime=end,\n", + " network_name=\"PERM:Alaska\",\n", + " session_name=\"A\",\n", + " system=\"G\",\n", + " field=[\"phase\", \"range\", \"snr\"],\n", + ")\n", + "await plan.plan()\n", + "print(plan) # preview the plan's request/group counts before any data is fetched\n", + "\n", + "\n", + "def summarize(table):\n", + " # Print a quick summary of one group's table.\n", + " d = pl.from_arrow(table)\n", + " stations = d[\"igs\"].unique().sort().to_list()\n", + " if len(stations) > 10:\n", + " stations = f\"{len(stations)} stations\"\n", + " print(len(d), d[\"timestamp\"].min(), d[\"timestamp\"].max(), stations)" + ] + }, + { + "cell_type": "markdown", + "id": "6f59fcbd-b77a-4733-82cc-b064b081bc66", + "metadata": {}, + "source": [ + "Printing a plan reports how many API requests and groups it will run before any data moves. This provides the users with a quick way to gauge how large a query is." + ] + }, + { + "cell_type": "markdown", + "id": "ac24a294-a48d-416a-a285-a2abd1f633c5", + "metadata": {}, + "source": [ + "Processing the plan one day at a time. You never hold more than a single day in memory at once." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "46d1af54-622f-4066-9895-cfd595341cdd", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_day():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "02c76245-1e7a-4663-b398-f0472a58eb68", + "metadata": {}, + "source": [ + "Processing the plan one station at a time across the whole window instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "34742255-bd16-4197-b151-1777b4753a1d", + "metadata": {}, + "outputs": [], + "source": [ + "async for table in plan.group_by_station():\n", + " summarize(table)" + ] + }, + { + "cell_type": "markdown", + "id": "187c7159-2923-4da7-adfd-d2bf3c41249c", + "metadata": {}, + "source": [ + " > **Check**: Each line of output represents one group." + ] + }, + { + "cell_type": "markdown", + "id": "8e88b724-65ec-4ef7-9e19-fb6bbf8c895b", + "metadata": {}, + "source": [ + "### A rough guide for choosing a strategy" + ] + }, + { + "cell_type": "markdown", + "id": "d0673a80-e5e9-4256-8f3f-a48ba2ddafa2", + "metadata": {}, + "source": [ + "* Less than a week: `.fetch()` the whole results at once.\n", + "* Weeks to months, or many stations: iterate with `group_by_day()` (or `group_by_station()`)\n", + "* Months to years: define custom batches with `plan.group_by()`." + ] + }, + { + "cell_type": "markdown", + "id": "6324d2fb-1711-4e13-b37c-bab35e7a4395", + "metadata": {}, + "source": [ + "For custom grouping, request ordering and performance details (concurrency, rate limiting, and retries), see the [EarthScope SDK documentation](https://docs.earthscope.org/sdk/query-plans#option-4-custom-grouping-advanced). The SDK applies no size limits by default. For very large queries, you can also cap memory or time (more information available in the same documentation)." + ] + }, + { + "cell_type": "markdown", + "id": "c5008800-7811-4fbc-9721-9869da927fc2", + "metadata": {}, + "source": [ + "## 7. Exploration Exercises\n", + "\n", + "Now that you've completed the core workflow, try modifying the parameters below to explore how the results change.\n", + "\n", + "**Try these modifications:**\n", + "\n", + "1. **Change the station:** Set `STATION` in the Configuration section to a different 4-character station ID and re-run Section 3. Does the station report the same constellations?\n", + "\n", + "2. **Isolate one signal:** In Section 5, request a single `obs_code` (e.g. `\"2L\"`) with `field=\"snr\"`. How much smaller is the result?\n", + "\n", + "3. **Save your results:** Write a fetched dataframe to the scratch directory (*Hint: Use `os.environ[\"SCRATCH_BUCKET\"]`*) as Parquet with `df.write_parquet(...)`, then read it back. Parquet preserves types and is far smaller than CSV." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "edf9b94a-b045-4fcf-be34-f2e2f958f124", + "metadata": {}, + "outputs": [], + "source": [ + "# Exploration cell — use this space to experiment" + ] + }, + { + "cell_type": "markdown", + "id": "1e147f5a-2352-4a16-b28b-3f5c1dbea902", + "metadata": {}, + "source": [ + "## 8. Troubleshooting & Support\n", + "\n", + "### Further Resources\n", + "\n", + "* [EarthScope SDK Documentation](https://docs.earthscope.org/sdk)\n", + "* [SDK GNSS Observation tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", + "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", + "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/tutorials/mvp-1-geodesy/NB2-instantaneous-PPP-positions.ipynb b/tutorials/mvp-1-geodesy/NB2-instantaneous-PPP-positions.ipynb new file mode 100644 index 0000000..de0f986 --- /dev/null +++ b/tutorials/mvp-1-geodesy/NB2-instantaneous-PPP-positions.ipynb @@ -0,0 +1,517 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2abe5d1d-4a2f-4c96-a78c-6c3e5f9ac750", + "metadata": {}, + "source": [ + "# Accessing Instantaneous Positions" + ] + }, + { + "cell_type": "markdown", + "id": "35320cb4-aed2-4420-8f9c-b3deb365e792", + "metadata": {}, + "source": [ + "**Version:** 1.0 | **Last updated:** 2026-07-14 | \n", + "\n", + "**Author:** Eshanta Mishra | **Author institution:** EarthScope Consortium\n", + "\n", + "**License:** CC-BY-4.0" + ] + }, + { + "cell_type": "markdown", + "id": "b9b1d1c3-569f-459a-8595-e1ba758e4bc7", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "**What this notebook does:** The notebook retrieves Instantaneous (high-rate PPP stream) GNSS positions as dataframes for selected stations and time ranges using the EarthScope SDK, and produces visualizations of the position streams and the displacement derived from them.\n", + "\n", + "**Why it is useful:** Instantaneous positions tell us where a station is right now, epoch by epoch. While daily solutions are useful for describing slow tectonic motion, this high-rate stream can capture ground movement that happens in seconds and provides a foundation for studying transient events like earthquakes.\n", + "\n", + "**What you will accomplish:** By the end, you will have retrieved instantaneous positions for one or more stations, inspected and understood data fields that come with the instantaneous positions, derived 2D and 3D displacement magnitudes, and visualized both the position streams and displacement over time.\n", + "\n", + "---\n", + "\n", + "### Prerequisites\n", + "\n", + "Before starting this notebook, you should:\n", + "\n", + "* [ ] Have completed: [Notebook 1 - Accessing GNSS Observations with the EarthScope SDK](NB1-access-gnss-via-SDK.ipynb).\n", + "* [ ] Be familiar with basic Python.\n", + "\n", + "---\n", + "\n", + "### GeoLab Compute Resources\n", + "\n", + "| Setting | Recommended |\n", + "|---|---|\n", + "| **Image** | GeoLab (default image) |\n", + "| **Server size** | 4 GB RAM, ~0.5 CPUs (default server) |" + ] + }, + { + "cell_type": "markdown", + "id": "811fc68e-1e81-482c-82dd-5efa2a89d715", + "metadata": {}, + "source": [ + "## Learning Objectives\n", + "\n", + "By the end of this notebook, you will be able to:\n", + "\n", + "1. Retrieve instantaneous (PPP) positions for one or more stations by station name, time range, facility, and software.\n", + "2. Load the result into a dataframe and interpret every returned field, including position components and their uncertainties.\n", + "3. Derive 2D and 3D displacement magnitudes from the east/north/up components.\n", + "4. Visualize position streams and displacement over time." + ] + }, + { + "cell_type": "markdown", + "id": "1ec3d79a-ad70-4c58-b8fc-287501adb7e9", + "metadata": {}, + "source": [ + "## Relevant Documentation & Resources\n", + "\n", + "* [EarthScope SDK documentation](https://docs.earthscope.org/sdk)\n", + "* [SDK GNSS Observations tutorial](https://docs.earthscope.org/sdk/gnss-obs-tutorial)\n", + "* [Polars User Guide](https://docs.pola.rs)\n", + "* [Altair (plotting)](https://altair-viz.github.io)" + ] + }, + { + "cell_type": "markdown", + "id": "076d400d-8c88-4113-a28d-91dac8648679", + "metadata": {}, + "source": [ + "## Contents\n", + "\n", + "1. [What are Instantaneous Positions?](#id-1-what-are-instantaneous-positions)\n", + "2. [Setup & Imports](#id-2-setup-imports)\n", + "3. [Retrieve Instantaneous Positions](#id-3-retrieve-instantaneous-positions)\n", + "4. [Inspect the Returned Fields](#id-4-inspect-the-returned-fields)\n", + "5. [Derive Displacement](#id-5-derive-displacement)\n", + "6. [Visualize Position Streams & Displacement](#id-6-visualize-position-streams-displacement)\n", + "7. [Exploration Exercises](#id-7-exploration-exercises)\n", + "8. [Troubleshooting & Support](#id-8-troubleshooting-support)" + ] + }, + { + "cell_type": "markdown", + "id": "f6ad6a72-e973-41aa-80f1-7b83114a8c18", + "metadata": {}, + "source": [ + "## 1. What are Instantaneous Positions?" + ] + }, + { + "cell_type": "markdown", + "id": "2b974c85-4624-4a92-94d8-6a097580d88f", + "metadata": {}, + "source": [ + "A GNSS station's position can be estimated in different ways depending on how much data goes into each estimate.\n", + "\n", + "A very common geodetic product is a **daily position time series** which provides one position per station per day, formed by combining a full 24 hours of observations. Averaging over a day suppresses the noise down to millimeter precision. Stacking those daily positions over years yields the velocity fields that reveal slow tectonic motion such as plate drift and interseismic strain at the level of a few millimeters per year. The tradeoff of this precision is the time resolution because one point per day cannot be used to show anything that happens in seconds.\n", + "\n", + "**Instantaneous positions** make the opposite tradeoff. Using Precise Point Positioning (PPP), the station's position is estimated epoch by epoch (once per second in this case). So you can see the current position of the stations continously. Individual estimates are far noisier than daily solutions. However, the high sampling rate makes it possible to detect rapid, transient motion in real time, such as an earthquake displacing a station within seconds. A daily solution would instead combine that motion into a single averaged position.\n", + "\n", + "> **Note:** Derived daily position time series are not yet available through the SDK; this notebook focuses on the instantaneous (high-rate PPP) stream, which is already available." + ] + }, + { + "cell_type": "markdown", + "id": "cca79b91-0348-4ee2-842e-740595288820", + "metadata": {}, + "source": [ + "## 2. Setup & Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8cd2903c-1b18-4877-a421-f19374f8fcf3", + "metadata": {}, + "outputs": [], + "source": [ + "# Standard library imports\n", + "import datetime as dt\n", + "\n", + "# Third-party imports\n", + "import altair as alt\n", + "import polars as pl\n", + "from earthscope_sdk import AsyncEarthScopeClient\n", + "\n", + "# Enable the Rust (vegafusion) backend so Altair can handle larger datasets efficiently\n", + "alt.data_transformers.enable(\"vegafusion\")\n", + "\n", + "es = AsyncEarthScopeClient()" + ] + }, + { + "cell_type": "markdown", + "id": "2448136f-5067-4c1f-98eb-96ea08b06ee4", + "metadata": {}, + "source": [ + "Polars' `.plot` accessor uses Altair under the hood, with vegafusion as its fast backend." + ] + }, + { + "cell_type": "markdown", + "id": "aa3e8c77-ceaf-4f6e-be2e-b99b0b7f0125", + "metadata": {}, + "source": [ + "### Configuration\n", + "\n", + "Set your parameters here before running the rest of the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15ecb388-d718-4949-a072-5db6cd398d65", + "metadata": {}, + "outputs": [], + "source": [ + "# Modify these values before running the notebook.\n", + "\n", + "STATIONS = [\"P146\", \"P147\", \"P148\"] # one or more station IDs\n", + "FACILITY = \"cwu\" # analysis center producing the stream\n", + "SOFTWARE = \"fastlane\" # PPP software producing the stream\n", + "META_FIELDS = [\"geosncl\"] # extra stream-metadata columns to attach\n", + "START = dt.datetime(2026, 6, 4, 10) # query start (UTC)\n", + "END = dt.datetime(2026, 6, 4, 12) # query end (UTC)" + ] + }, + { + "cell_type": "markdown", + "id": "c1c52777-0410-4a6f-b3e6-198cfeebb318", + "metadata": {}, + "source": [ + "## 3. Retrieve Instantaneous Positions\n", + "\n", + "In this step, we will retrieve high-rate PPP position stream for one or more stations, returned as an Apache Arrow table.\n", + "\n", + "As with observations on [Notebook 1](NB1-access-gnss-via-SDK.ipynb), the SDK returns Arrow, which converts into a dataframe with little or no copying. The stream is selected not just by station and time, but also by the processing pipeline that produced it such as the analysis `facility` and the `software`.\n", + "\n", + "Each argument below narrows what you get:\n", + "\n", + "* `station_name`: one station ID or a list\n", + "* `facility`: the analysis center producing the stream (here `\"cwu\"`)\n", + "* `software`: the PPP engine (here `\"fastlane\"`)\n", + "* `meta_fields`: extra stream-metadata columns to attach (here `\"geosncl\"`, the stream identifier)\n", + "\n", + "The expected result is roughly 21,600 rows, i.e. 3 stations x 2 hours x 3600 seconds, since the stream is 1 Hz." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "475dd2e2-9db8-453d-b495-d6afa1417d8b", + "metadata": {}, + "outputs": [], + "source": [ + "# Fetch a 2-hour window of 1 Hz instantaneous positions for the selected stations.\n", + "table = await es.data.gnss_instantaneous_positions(\n", + " start_datetime=START,\n", + " end_datetime=END,\n", + " station_name=STATIONS,\n", + " facility=FACILITY,\n", + " software=SOFTWARE,\n", + " meta_fields=META_FIELDS,\n", + ").fetch()\n", + "\n", + "df = pl.from_arrow(table).sort(\"timestamp\")\n", + "print(f\"{len(df):,} rows\")\n", + "df.head()" + ] + }, + { + "cell_type": "markdown", + "id": "6069a914-e056-472a-a968-516aff9acf58", + "metadata": {}, + "source": [ + "## 4. Inspect the Returned Fields\n", + "\n", + "Get to know the data before analyzing it: which columns came back, how often the stream samples, which stations are present, and where values are missing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "161efa31-b733-4d24-a5de-595b688bf9ae", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"Columns:\", df.columns)\n", + "print(\"Streams:\", df[\"geosncl\"].unique().sort().to_list())" + ] + }, + { + "cell_type": "markdown", + "id": "dbacb342-cda9-4a12-9a58-b47d8e487342", + "metadata": {}, + "source": [ + "### Sampling Interval" + ] + }, + { + "cell_type": "markdown", + "id": "0d4d0465-7f19-43e8-a344-73060dc3ab28", + "metadata": {}, + "source": [ + "Look at the spacing between consecutive epochs for a single stream. It should be 1 seconds." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "92c6c91f-c237-4164-98f9-bae98288a0e0", + "metadata": {}, + "outputs": [], + "source": [ + "one = df.filter(pl.col(\"geosncl\").str.starts_with(STATIONS[0])).sort(\"timestamp\")\n", + "one.select(pl.col(\"timestamp\").diff().alias(\"dt\"))[\"dt\"].drop_nulls().value_counts(sort=True).head()" + ] + }, + { + "cell_type": "markdown", + "id": "c7a92d10-ec53-4cc8-80d2-2909ad699ade", + "metadata": {}, + "source": [ + "### Missing values" + ] + }, + { + "cell_type": "markdown", + "id": "0482145d-6c24-48b4-ab4a-20439d071593", + "metadata": {}, + "source": [ + "Some epochs may lack a horizontal solution, so `east` / `north` can be null even when `up` is present. Count nulls per column. This matters when we derive displacement below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dbfeb030-1f33-4c71-bb81-2cc45ded6ffc", + "metadata": {}, + "outputs": [], + "source": [ + "df.null_count()" + ] + }, + { + "cell_type": "markdown", + "id": "e4ab8121-60a6-4f03-8474-9939434fa14f", + "metadata": {}, + "source": [ + "### What each field means" + ] + }, + { + "cell_type": "markdown", + "id": "d50450ea-b0cb-4398-9f31-30073cb18ab2", + "metadata": {}, + "source": [ + "| Column | Type | Meaning |\n", + "|---|---|---|\n", + "| `timestamp` | datetime (UTC) | Epoch of the position estimate. The stream is 1 Hz, i.e. one row per second, per station. |\n", + "| `east` | float | East displacement from the station's reference position, in meters. |\n", + "| `north` | float | North displacement, in meters. |\n", + "| `up` | float | Vertical (up) displacement, in meters. |\n", + "| `sig_ee` | float | 1-sigma uncertainty on `east`, in meters. |\n", + "| `sig_nn` | float | 1-sigma uncertainty on `north`, in meters. |\n", + "| `sig_uu` | float | 1-sigma uncertainty on `up`, in meters. |\n", + "| `q_channel` | int | Integer-encoded quality and processing status information for each position estimate. |\n", + "| `ingest_latency` | duration | Time between the observation epoch and its ingest by the server. |\n", + "| `processing_delay` | duration | Time taken to process the epoch. |\n", + "| `geosncl` | str | Compound stream identifier: station.network.channel-location (e.g. `P146.PW.LY_.00`). |" + ] + }, + { + "cell_type": "markdown", + "id": "b3f044a4-6ee3-4aba-a4c9-74793839fd72", + "metadata": {}, + "source": [ + "## 5. Derive Displacement\n", + "\n", + "The position components combine into a single displacement magnitude which tells us how far the station sits from its reference position at each epoch. Horizontal (2D) uses east and north; total (3D) adds the vertical:\n", + "\n", + "$$d_{2D} = \\sqrt{east^2 + north^2} \\qquad d_{3D} = \\sqrt{east^2 + north^2 + up^2}$$" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "42de7c31-9908-4286-92fb-4afa03f5f297", + "metadata": {}, + "outputs": [], + "source": [ + "disp = df.with_columns(\n", + " (pl.col(\"east\").pow(2) + pl.col(\"north\").pow(2)).sqrt().alias(\"disp_2d\"),\n", + " (pl.col(\"east\").pow(2) + pl.col(\"north\").pow(2) + pl.col(\"up\").pow(2)).sqrt().alias(\"disp_3d\"),\n", + ")\n", + "\n", + "disp.select([\"timestamp\", \"geosncl\", \"east\", \"north\", \"up\", \"disp_2d\", \"disp_3d\"]).head()" + ] + }, + { + "cell_type": "markdown", + "id": "efe69ab7-0e85-4692-a375-6dfd6a87fa48", + "metadata": {}, + "source": [ + "Note that the square root propagates nulls: if `east` or `north` is missing for an epoch, that epoch's `disp_2d` and `disp_3d` are null too. Drop them before computing statistics if needed using:\n", + "\n", + "```python\n", + "disp_clean = disp.drop_nulls(subset=[\"disp_2d\"])\n", + "```\n", + "\n", + "> **Check:** For a quiet station, horizontal displacement (`disp_2d`) should stay small and roughly steady — a few centimeters of PPP scatter around the reference. A sudden *step* in this value over time is what ground motion (e.g. an earthquake) would look like." + ] + }, + { + "cell_type": "markdown", + "id": "7a66e9a7-c8c7-4b1d-91da-0e4471397815", + "metadata": {}, + "source": [ + "## 6. Visualize Position Streams & Displacement\n", + "\n", + "### The position components over time\n", + "\n", + "Reshape one station's `east` / `north` / `up` into long form and plot them together to see the three streams at once." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "538f9587-fa9f-4e50-a746-b8d2b9c28f25", + "metadata": {}, + "outputs": [], + "source": [ + "one_long = df.filter(pl.col(\"geosncl\").str.starts_with(STATIONS[0])).unpivot(\n", + " [\"east\", \"north\", \"up\"],\n", + " index=\"timestamp\",\n", + " variable_name=\"component\",\n", + " value_name=\"meters\",\n", + ")\n", + "\n", + "one_long.plot.line(x=\"timestamp\", y=\"meters\", color=\"component\").properties(\n", + " width=800, height=300, title=f\"{STATIONS[0]}: position components over time\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "217c22a5-5f14-4bad-81c3-4a64e9bafae6", + "metadata": {}, + "source": [ + "### Displacement over time\n", + "\n", + "Plot the 2D horizontal displacement for every station together. A quiet station traces a roughly flat, noisy band; a step would signal real motion." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c5c5632d-72c5-41fe-8484-37acb86f7091", + "metadata": {}, + "outputs": [], + "source": [ + "disp.plot.line(x=\"timestamp\", y=\"disp_2d\", color=\"geosncl\").properties(\n", + " width=800, height=300, title=\"2D horizontal displacement over time\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "6aa31eda-ecd7-490d-8438-82b45706b732", + "metadata": {}, + "source": [ + "### A note on stream timeliness\n", + "\n", + "The `ingest_latency` and `processing_delay` columns describe how *timely* the stream is, not how accurate the positions are. They are Polars **Duration** types — and Altair can only plot **numeric** axes, so you must convert a Duration to a number (milliseconds) before plotting. This is a common trap." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7d463152-2785-453e-97d4-6f52f5e4e380", + "metadata": {}, + "outputs": [], + "source": [ + "# Altair plots only numeric axes, so convert the Duration column to milliseconds first.\n", + "df.with_columns(\n", + " pl.col(\"ingest_latency\").dt.total_milliseconds().alias(\"ingest_latency_ms\")\n", + ").plot.line(x=\"timestamp\", y=\"ingest_latency_ms\", color=\"geosncl\").properties(\n", + " width=800, height=300, title=\"Ingest latency (ms)\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "c5008800-7811-4fbc-9721-9869da927fc2", + "metadata": {}, + "source": [ + "## 7. Exploration Exercises\n", + "\n", + "Try modifying the parameters to explore how the results change.\n", + "\n", + "1. **Different stations or window:** Change `STATIONS` and the `START`/`END` window in Configuration and re-run. Do all stations return a stream for your window?\n", + "2. **Show the uncertainty:** Plot `up` for one station with a shaded band of +/- `sig_uu` around it (hint: Altair's `mark_area` or `mark_errorband`). How wide is the vertical uncertainty compared to the signal?\n", + "3. **Noisiest station:** Compute the per-station standard deviation of `disp_2d` and identify which station is noisiest." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "edf9b94a-b045-4fcf-be34-f2e2f958f124", + "metadata": {}, + "outputs": [], + "source": [ + "# Exploration cell — use this space to experiment" + ] + }, + { + "cell_type": "markdown", + "id": "1e147f5a-2352-4a16-b28b-3f5c1dbea902", + "metadata": {}, + "source": [ + "## 8. Troubleshooting & Support\n", + "\n", + "### Common Issues\n", + "\n", + "| Error | Likely cause | Fix |\n", + "|---|---|---|\n", + "| Plotting error on a duration column | Altair cannot plot Duration types directly | Convert with `.dt.total_milliseconds()` before plotting |\n", + "| Null `disp_2d` / `disp_3d` values | `east` or `north` was null for that epoch | Use `drop_nulls(subset=[\"disp_2d\"])` before computing statistics |\n", + "\n", + "### Further Resources\n", + "\n", + "* [EarthScope SDK Documentation](https://docs.earthscope.org/sdk)\n", + "* [GeoLab Documentation](https://docs.earthscope.org/geolab)\n", + "* [GeoLab Community Forum](https://earthscope.discourse.group/latest)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}