diff --git a/advanced_topics/env_mgmt.md b/advanced_topics/env_mgmt.md deleted file mode 100644 index 88d3b27..0000000 --- a/advanced_topics/env_mgmt.md +++ /dev/null @@ -1,71 +0,0 @@ -# Environment Management - -When you open GeoLab and select an environment when starting your server, you launch into a pre-configured virtual python environment. These environments typically include many commonly used geophysics tools, but we cannot include everything. Additional software and packages can be installed by users in GeoLab, with several options: - -* [Ephemeral Installation:](#ephemeral-installation) Due to the ephemeral nature of the JupyterHub servers, packages installed on the server will only be valid for your current session and will not persist from one session to another. You will need to reinstall them each time you log back in to GeoLab, or add them to your notebooks as magic commands. -* [Create a Custom Image:](#create-a-custom-image) If you need to use the same software repeatedly, during multiple sessions, or for multiple users, you can create your own software environment and load it at launch time. -* [Bring Your Own Image:](#bring-your-own-image) Many other organizations (e.g, NASA, NOAA) maintain their own JupyterHub compute environment images. Many of these will run in GeoLab. - -## Ephemeral Installation -### Installing python packages - Use `pip` or `conda` to install the package yourself at the beginning of your session by typing the below in a GeoLab Terminal. -``` -pip install pkgname -conda install pkgname -``` - -Because installations are ephemeral, we recommend adding these commands to your python notebooks using a magic command. This will allow the python notebook to run the bash command, and ensure you re-install the right packages each time you return to your project without having to remember to type it in the terminal. - -You can add a magic command to the beginning of your notebook. The `%pip` / `%conda` forms here are better than their `!pip` / `!conda` counterparts to ensure that the package is installed to the correct directory. - -``` -%pip install pkgname -``` - -### Installing from a requirements file - -If you have a `requirements.txt` or `environment.yml` file (e.g., from a cloned repository), you can install all listed packages at once. - -**From a `requirements.txt` file:** -```bash -%pip install -r requirements.txt -``` - -**From a `environment.yml` file:** -```bash -%conda env update --file environment.yml --name base -``` - -Note that `--name base` installs packages into the active base environment rather than creating a new one, which is necessary in GeoLab's ephemeral setup. As with individual package installs, these will not persist after your session ends. - -We have found that in GeoLab, pip is often faster, while conda's dependency solver is more robust. If you encounter dependency conflict while installing with pip, try conda. - -```{{Note}} -A Note on reproducible conda environments: -When you launch GeoLab, you are already inside a conda environment. If you create a _new_ conda environment, you can maintain the list of packages (ie, in an environment.yml), but because of the ephemeral nature of the you will have to re-install them all in GeoLab each session. In order to use the new environment to run a notebook, you will need to install the environment to a kernel. For example, to create a kernel called 'Seis', you would need to run the following line of code in the terminal: - -`jupyter ipykernel install --user --name Seis --display-name "Seis" ` - -and then select the new kernel from the Kernel menu or from the dropdown of available kernels in the upper right of the notebook toolbar. - -``` - -### Installing ubuntu packages -Some software can be installed via the terminal using `apt-get`. Please be advised that users do not have the ability to run commands as super users: `sudo` will not work and these permissions _cannot_ be granted to individual users. If you need to install something foundational, please create a custom image. - -### C/C++/Fortran Compilers -Most GeoLab images have `gcc`, `g++`, and `gfortran` compilers installed. You will need to move your makefiles into your home directory and compile them within GeoLab. - -## Create a Custom Image -:::{attention} Section Under Development! -::: - -EarthScope is working to build more comprehensive documentation on custom image creation. In the mean time, please see: - -- [2i2c guide to binderhub](https://docs.2i2c.org/user/environment/dynamic-imagebuilding/#user-environment-building) - This strategy is an option if you want to use an environment file that extends the base GeoLab image with additional python packages, but don't want to re-install it every time (i.e., with an environment file). You can specify your environment file once to generate a stable container, and load that at launch time. Note, there's a 15-20 minute overhead to build the image in binderhub the first time and any time you change the environment file, so you'll need to consider whether this will save you time in the long run. - -- If you need to make extensive software changes, you can build your own image using our existing image as a template. Instructions are located in the `README.md` file in the [GeoLab GitHub repository.]({{ geolab_github }}) You will need to clone the repository, modify the build files to suit your needs, build your container and upload it to a public container registry like DockerHub or AWS ECR, and then select it as 'Other' from the dropdown when you [launch your GeoLab server](../getting_started/server_launch.md). This strategy is an advanced approach for software with complex installations, and only recommended for users with some prior knowledge of building docker containers. - -## Bring Your Own Image -Many other organizations (e.g., NASA, NOAA, Pangeo, The Rocker Project, etc) maintain JupyterHub images with their software installed. Many of these will run in GeoLab (but we make no guarantees that images built for other platforms will work here!) We recommend using these images interdisciplinary research, where you may need software installed in another image, but want to run it on data stored in the NSF NGF archive. At [Server Launch](../getting_started/server_launch.md#select-an-environment), select the 'Other' option from the Environment dropdown and enter the URL to the public image. - \ No newline at end of file diff --git a/advanced_topics/environments/binder_for_images.md b/advanced_topics/environments/binder_for_images.md new file mode 100644 index 0000000..5fecf84 --- /dev/null +++ b/advanced_topics/environments/binder_for_images.md @@ -0,0 +1,182 @@ +# GeoLab Binder + +There are two ways to launch a custom environment in GeoLab. The first is to build a Docker image, push it to a registry, and paste the image URL. The second, called **GeoLab Binder**, is to point GeoLab at a GitHub repository and let it build the environment for you. This guide covers how the second approach works and when to use it. + +--- + +## What Is GeoLab Binder? + +GeoLab Binder lets you launch a session from a GitHub repository instead of a prebuilt image. You put configuration files in your repo that describe what packages you need, and GeoLab builds the environment automatically when you start a session. + +Behind the scenes, GeoLab reads the Dockerfile and configuration files in your repository and builds the container image for you. You write the Dockerfile (using the same template as a custom image), but you never run Docker locally or push to a registry, as GeoLab handles the build. + +Both approaches run inside GeoLab, so you have the same access to EarthScope data services either way. + +```text +GitHub repo with config files → GeoLab builds image → Session launches in GeoLab +``` + +--- + +## GeoLab Binder vs. Custom Image + +Both approaches use GeoLab's environment selector, but they use different options. The difference is what you select and what you paste. + +| | **GeoLab Binder** | **Custom Image** | +| --- | --- | --- | +| Menu option | **Build your own image** | **Other** | +| What you paste | A GitHub repo URL | A container registry URL (`ghcr.io/...`) | +| Who builds the image | GeoLab, automatically from your repo | You, locally with Docker | +| What you need | Dockerfile + config files in a GitHub repo | Docker Desktop, local build, registry push | +| EarthScope data access | Yes | Yes | +| How updates work | Push a commit, and the next launch picks up the changes | Rebuild and push a new image tag | +| Best for | Getting started quickly, iterating, sharing a repo | Stable, versioned environments; full Docker control | + +**Use GeoLab Binder when** you want to share a GitHub repository as a runnable environment without running Docker locally or managing a container registry. + +**Use a custom image when** you need a fixed, versioned environment that won't change between sessions, or when you need more control than config files allow. See [Building a Custom GeoLab Image](./building_custom_images.md). + +--- + +## Before You Start + +You need: + +- A **GitHub account** and a **GitHub repository** for your project. +- A GeoLab account at [GeoLab](https://geolab.earthscope.cloud). + +No local Docker builds or container registry required. + +--- + +## Step 1: Set Up Your GitHub Repository + +Create a new repository on GitHub (or use an existing one). Add your notebooks and at least one configuration file: + +```text +my-project/ +├── Dockerfile ← required (copy from the geolab-base template) +├── environment.yml ← your packages (conda) +├── requirements.txt ← PyPI-only packages (optional) +├── apt.txt ← system software (optional, rarely needed) +├── postBuild ← setup commands to run after install (optional) +└── my_notebook.ipynb ← your notebooks +``` + +Copy the `Dockerfile` from the `geolab-base` template at [EarthScope/GeoLab](https://github.com/EarthScope/GeoLab/tree/main/geolab-base) and leave it unchanged. All your customization goes in the other files. GeoLab Binder reads everything in the repo and builds the image when you launch. + +--- + +## Step 2: Write Your Configuration Files + +### `environment.yml`: Your Main Package List + +```yaml +name: my-binder-env +channels: + - conda-forge +dependencies: + - python=3.11 + - numpy + - matplotlib + - obspy +``` + +### `requirements.txt`: PyPI-only Packages (If Needed) + +```text +some-pypi-package==1.2.3 +``` + +### `apt.txt`: System Software (If Needed) + +```text +build-essential +``` + +### `postBuild`: One-time Setup Commands (Optional) + +If you need to run something after packages install, such as downloading a data file or enabling a Jupyter extension, create a `postBuild` file: + +```bash +#!/bin/bash +set -e +jupyter labextension install my-extension +``` + +Before committing, mark it executable: + +```bash +chmod +x postBuild +git add postBuild +git commit -m "add postBuild script" +``` + +These files use the exact same format as the custom image template. If you've worked through [Building a Custom GeoLab Image](./building_custom_images.md), you can reuse them directly. + +Once your files are ready, commit and push them to GitHub: + +```bash +git add environment.yml requirements.txt apt.txt +git commit -m "add environment configuration" +git push +``` + +If this is a new repository, you may need to set the upstream branch on your first push: + +```bash +git push -u origin main +``` + +--- + +## Step 3: Launch from Your Repo in GeoLab + +1. Go to [earthscope.org/data/geolab](https://www.earthscope.org/data/geolab/) and click **Launch GeoLab**. +2. Enter your username and password, then click **Continue**. +3. In the environment selector, choose **Build your own image**. +4. In the **Repository** field, paste the git clone URL for your repository: + + ```text + https://github.com/your-github-username/my-project.git + ``` + +5. Click **Build Image**. GeoLab will build the environment from your repository. This can take several minutes the first time. +6. When you receive a notification that the build is complete, click **Start**. + +> **If the build fails:** Check the build log for error messages. Common causes are a misspelled package name, an unavailable package version, or a syntax error in `environment.yml`. Fix the file, push the commit, and try again. + +--- + +## Step 4: Share Your Environment + +To share your environment with someone else, give them the git clone URL for your repository. They follow the same steps: go to GeoLab, choose **Build your own image**, paste the URL, and launch. + +Because the environment is defined by files in the repo, anyone who uses the same URL gets the same packages. If you update `environment.yml` and push, the next person to launch picks up the new version automatically. + +--- + +## Updating Your Environment + +Edit your configuration files, commit, and push. The next GeoLab launch from that repo URL will rebuild with the updated packages. + +```bash +# edit environment.yml, then: +git add environment.yml +git commit -m "add pandas to environment" +git push +``` + +> **Note:** A new commit triggers a fresh build on the next launch. If you're changing packages frequently, expect slower first launches while GeoLab rebuilds the image. + +--- + +## Quick Reference + +| What you want to do | How | +| --- | --- | +| Define your packages | `environment.yml` (conda), `requirements.txt` (PyPI) | +| Launch from a repo | GeoLab → Start Server → Build your own image → paste git clone URL | +| Share your environment | Share the git clone URL (`https://github.com/username/repo.git`) | +| Update your environment | Edit config files, commit, and push | +| Use a fixed versioned environment instead | See [Building a Custom GeoLab Image](./building_custom_images.md) | \ No newline at end of file diff --git a/advanced_topics/environments/building_custom_images.md b/advanced_topics/environments/building_custom_images.md new file mode 100644 index 0000000..2eb5746 --- /dev/null +++ b/advanced_topics/environments/building_custom_images.md @@ -0,0 +1,309 @@ +# Building a Custom GeoLab Image + +A GeoLab image is a complete, prepackaged computing environment that runs in JupyterLab. It includes common geophysics Python and scientific packages bundled together. Starting a GeoLab session launches an image. This step-by-step guide walks through building an image with a customized environment. + +--- + +## How It Works + +Think of an **image** as a recipe and a **container** as a meal made from that recipe. The recipe doesn't change; you can make the same meal over and over. GeoLab does the same thing: it takes your image and launches a fresh session from it every time. + +Install Python packages in an image by editing plaintext files that list the required software. A tool called `Docker` reads those files and builds the image. The image must be published in an image repository so GeoLab can access it. Here is the process: + +```text +Edit config files → Docker builds → Image → Push image to repository → GeoLab runs it +``` + +--- + +## Before Starting + +Two pieces of software must be installed on **your computer**: + +1. **Docker Desktop.** Download it at [docker.com](https://www.docker.com/products/docker-desktop/). Install it on your computer, open it, and leave it running in the background. +2. **Git client**, to download the GeoLab Dockerfile template. Use the operating system's package manager to install a git client. + +Verify Docker and git are installed and working by opening a terminal and running: + +```bash +docker --version +git --version +``` + +If they print a version number, they are installed and working. + +In addition to the required software, a **GitHub** account at [github.com](https://github.com), a Docker account, or an AWS account is needed for publishing the image and making it available for GeoLab. + +--- + +## Step 1: Get the Template + +EarthScope provides a starter template. Download it using git to set up a working folder: + +```bash +git clone --depth 1 https://github.com/EarthScope/GeoLab.git +cp -R GeoLab/geolab-base my-geolab-image +cd my-geolab-image +``` + +The `my-geolab-image` folder contains these files: + +```text +my-geolab-image/ +├── Dockerfile ← do not edit this +├── environment.yml ← add your conda packages here +├── requirements.txt ← add PyPI-only packages here +├── apt.txt ← add system software here (rarely needed) +├── start ← do not edit this +├── test_packages.py ← smoke test for installed packages +└── test_notebook.ipynb ← interactive version of the smoke test +``` + +> **The only files to edit are `environment.yml`, `requirements.txt`, and `apt.txt`.** Everything else is set up for you. + +--- + +## Step 2: Add Your Packages + +### `environment.yml`: Your Main Package List + +Add conda Python packages from `conda-forge` here. Open the file and add packages under the `dependencies` section: + +```yaml +channels: + - conda-forge + - nodefaults +dependencies: + - python=3.12 + # --- Geophysics --- + - obspy + - pygmt + # --- Geospatial --- + - cartopy + - geopandas + # add your packages below: + - my-package-name +``` + +Conda packages are preferred, because conda checks that everything works together before installing and reduces the possibility of dependency conflicts among packages. + +### `requirements.txt`: Packages Only on PyPI + +Some packages aren't available through conda-forge and must be installed from PyPI. Add them here, one per line: + +```text +earthscope-sdk==1.4.1 +seisbench +``` + +### `apt.txt`: System Software (Rarely Needed) + +Most scientific packages go in `environment.yml`. Only use `apt.txt` for low-level system tools that can't be installed any other way: + +```text +build-essential +git +``` + +--- + +## Step 3: Build and Test Locally + +Before publishing, build the image on your computer and make sure everything works. + +**Build the image:** + +```bash +docker build -f Dockerfile --tag my-geolab-image:0.1.0 . +``` + +This reads the config files and assembles the image. It can take several minutes the first time. + +**Run it locally:** + +```bash +docker run --rm -p 8888:8888 my-geolab-image:0.1.0 +``` + +Look in the output for a line like: + +```text +http://127.0.0.1:8888/lab?token=... +``` + +Copy that URL into a browser and a JupyterLab session will open. + +**Optional: Test that your packages installed correctly:** + +In the JupyterLab session, test if a package was installed and functions as expected. A simple version check is often sufficient to determine if a package has been installed and is working. + +To test a package in a notebook environment, open a notebook and add this code to a cell: + +```python +import importlib +import shutil +import subprocess +import sys + +RESULTS = [] + + +def py(modname, alias=None, smoke=None): + """Import `modname` and optionally run `smoke(mod)` as a sanity check.""" + label = alias or modname + try: + mod = importlib.import_module(modname) + if smoke is not None: + smoke(mod) + version = getattr(mod, '__version__', '') + RESULTS.append((label, 'OK', str(version), '')) + except Exception as exc: + RESULTS.append((label, 'FAIL', '', f'{type(exc).__name__}: {exc}')) + + +def cli(cmd, version_flag='--version'): + """Verify `cmd` is on $PATH and responds to a version flag.""" + path = shutil.which(cmd) + if not path: + RESULTS.append((cmd, 'FAIL', '', 'not on $PATH')) + return + try: + r = subprocess.run([cmd, version_flag], + capture_output=True, text=True, timeout=10) + line = (r.stdout or r.stderr).strip().splitlines() + version = line[0] if line else 'on PATH' + RESULTS.append((cmd, 'OK', version[:80], '')) + except Exception as exc: + RESULTS.append((cmd, 'OK', 'on PATH', f'{type(exc).__name__}')) + + +print(f'Python {sys.version}') +print(f'sys.prefix: {sys.prefix}') +``` + +Open another cell and use either the `py` or `cli` function to test a package. For example: + +```python +py('earthscope_sdk', alias='earthscope-sdk') +cli('es') # earthscope-cli entry point +``` + +Open a new cell and add the following code to summarize the results of the test: + +```python +import pandas as pd +from IPython.display import display + +df = pd.DataFrame(RESULTS, + columns=['package', 'status', 'version', 'error']) + +passed = int((df['status'] == 'OK').sum()) +total = len(df) +failed = total - passed + +print(f'Results: {passed}/{total} OK, {failed} failed') +if failed: + print('\nFailures:') + for _, row in df[df['status'] == 'FAIL'].iterrows(): + print(f" {row['package']:35s} {row['error']}") + +df.style.map( + lambda v: ('color: red; font-weight: bold' if v == 'FAIL' + else 'color: green'), + subset=['status'] +) +``` + +Each package gets a pass or fail. If something fails, it usually means a package name is misspelled or a version is unavailable, so go back to `environment.yml` and fix it, then rebuild. + +--- + +## Step 4: Publish Your Image to Docker Hub + +Once the local test passes, rebuild the image for GeoLab's platform and push it to a container registry. + +**Rebuild for GeoLab's platform:** + +GeoLab uses linux/amd64 images. Depending on your computer operating system (Windows or macOS), you may have to rebuild the image for the linux/amd64 platform. In addition, the image must be published in an image repository that is accessible to GeoLab. + +If you created an account using Docker Desktop, pushing an image to Docker Hub (the Docker image repository) does not require additional authentication. `Tag` or name the image with your Docker username and the name of the image. + +```bash +docker build --no-cache -f Dockerfile \ + --platform linux/amd64 \ + --tag your-docker-username/my-geolab-image:0.1.0 . +``` + +> **Why `--platform linux/amd64`?** GeoLab runs on Linux. If you're on a Mac with Apple Silicon, your local machine uses a different architecture. This flag ensures the image works on GeoLab regardless of what you built it on. + +**Push the Image to Docker Hub:** + +```bash +docker push your-docker-username/my-geolab-image:0.1.0 +``` + +By default, images published to Docker Hub are public and available for use with GeoLab. + +### Publishing to GitHub or AWS Image Repositories + +Alternatives to Docker Hub include GitHub Container Registry (ghcr) or AWS Elastic Container Registry (ECR). Choosing an image repository depends on user requirements. GitHub features a tight integration with CI (Continuous Integration) through GitHub Actions that can trigger an image build and push to ghcr. This automates the process of building and pushing an image through a `pull request`. AWS ECR offers cloud-scale uploads and downloads to support multiple instances of GeoLab requested by hundreds of users or more. + +Both ghcr and ECR have more stringent authorization practices and controls over publicly available images. For a step-by-step walkthrough for pushing images to either repository, go to [Pushing Images to GitHub or AWS ECR](./pushing_to_ghcr_ecr.md) for detailed instructions. + +--- + +## Step 5: Launch It in GeoLab + +1. Go to [earthscope.org/data/geolab](https://www.earthscope.org/data/geolab/) and click **Launch GeoLab**. +2. Enter your username and password, then click **Continue**. +3. If a **Stop My Server** button appears, click it first. +4. Click **Start My Server**. +5. Under **Environment**, choose **Other**. +6. In the **Custom image** field, enter your image name, e.g. `ghcr.io/your-github-username/my-geolab-image:0.1.0`. +7. Click **Start**. + +GeoLab will pull your image and launch a session from it. The first launch takes a minute while it downloads; after that it's cached and starts quickly. + +--- + +## Making Changes Later + +Edit your config files, then rebuild and push with a new version number: + +```bash +docker build --no-cache -f Dockerfile \ + --platform linux/amd64 \ + --tag ghcr.io/your-github-username/my-geolab-image:0.1.1 . + +docker push ghcr.io/your-github-username/my-geolab-image:0.1.1 +``` + +> **Always use a new version number** (`0.1.1`, `0.1.2`, etc.) when you rebuild. If you reuse the same tag, GeoLab may load the old cached version instead of your new one. + +--- + +## Quick Reference + +| What you want to do | Where to do it | +| --- | --- | +| Add a Python package | `environment.yml` under `dependencies` | +| Add a PyPI-only package | `requirements.txt` | +| Add a system tool | `apt.txt` | +| Build locally for testing | `docker build --tag my-geolab-image:0.1.0 .` | +| Run locally | `docker run --rm -p 8888:8888 my-geolab-image:0.1.0` | +| Test packages | Run the test cells in a notebook (see Step 3) | +| Build for GeoLab | `docker build --no-cache --platform linux/amd64 --tag ghcr.io/username/image:version .` | +| Publish | `docker push ghcr.io/username/my-geolab-image:0.1.0` | + +--- + +## Getting a Personal Access Token (for GHCR) + +Before you can push images to GHCR, you need a **Personal Access Token (PAT)** with package permissions: + +1. Go to **GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic)**. +2. Click **Generate new token (classic)**. +3. Give it a name (e.g. `geolab-image`), set an expiration, and check the **`write:packages`** scope. +4. Click **Generate token** and copy it, as you won't be able to see it again. + +Save your token somewhere safe (a password manager works well). You'll use it to log in to the registry in Step 4. \ No newline at end of file diff --git a/advanced_topics/environments/index.md b/advanced_topics/environments/index.md new file mode 100644 index 0000000..beacedd --- /dev/null +++ b/advanced_topics/environments/index.md @@ -0,0 +1,37 @@ +# Managing Environments in GeoLab + +When you open GeoLab and select an environment, you launch a pre-configured virtual Python environment that includes commonly used geophysics software. You can install additional software and packages in GeoLab, using one of several options: + +## [Ephemeral Installation:](#ephemeral-installation) + +Environments in JupyterHub are temporary. Packages installed while using GeoLab are only valid during the current session and will not persist from one session to the next. Packages must be reinstalled every time a new GeoLab instance is launched. For ephemeral installs, add packages using [line magic](https://ipython.readthedocs.io/en/8.24.0/interactive/magics.html#line-magics) commands. + +Because installations are temporary, we recommend adding these commands to the beginning of a notebook. This ensures the required packages are installed and available each time the notebook runs. + +Use the `%pip` / `%conda` magic commands because they execute in the active Python notebook kernel. The `!` magic command executes in the system shell and is best used for running CLI tools or file-system checks. + +```ipynb +%pip install pkgname +``` + +For more information about managing environments, go to [`Managing Environments`](./managing_environments.md) for detailed instructions. + +## [Build a Custom Image:](#build-a-custom-image) + +If your task or analysis requires the same software over multiple sessions or for multiple users, building a custom image with the required packages is best practice. + +Building a custom image requires software such as Docker to build the images. GeoLab can use custom images available on public image repositories such as Docker Hub, GitHub Container Registry (ghcr), or AWS Elastic Container Registry (ECR). + +For more information about building a custom image, go to [`Building Custom Images`](./building_custom_images.md) for detailed instructions on building and pushing images to an image repository. + +## [Run an Image From a GitHub Code Repository With Binder](#run-an-image-from-a-github-code-repository-with-binder) + +GeoLab can run environments from a GitHub repository using [Binder](https://mybinder.readthedocs.io/en/latest/). The process is similar to building a custom image, but instead of building and storing the image with Docker, you provide the URL of the GitHub repository to GeoLab and the image will be dynamically built and opened. + +While this is the simplest method for building and running a custom image, GeoLab builds the image every time. The build process can take some time and is not ideal for multiple users. By contrast, an image in a repository is pulled directly into JupyterHub and started without waiting for it to be built. + +For more information about using Binder, go to [`Binder for Images`](./binder_for_images.md) for detailed instructions on building a repository that works with Binder. + +## [Bring a JupyterHub Image](#bring-a-jupyterhub-image) + +Other organizations, such as NASA or NOAA, maintain their own JupyterHub compute environment images that run in GeoLab. Using an image from another organization uses the same process as a custom image. Select `Other...` in the Environment pull-down menu, enter the image URL, select the `Resource Allocation`, and select `Start`. diff --git a/advanced_topics/environments/managing_environments.md b/advanced_topics/environments/managing_environments.md new file mode 100644 index 0000000..94a8cdb --- /dev/null +++ b/advanced_topics/environments/managing_environments.md @@ -0,0 +1,177 @@ +# Creating Your Own Python Environment in GeoLab + +Python notebooks use packages: collections of reusable code that perform tasks like processing data, making maps, or analyzing signals. A notebook may need packages that aren't available at run time. An **environment** is a way to keep all of those packages organized in one tidy, self-contained workspace. + +This guide shows you how to create your own environment in GeoLab. + +--- + +## What Is a Package Manager? + +GeoLab uses a tool called **conda** to install and manage packages. Think of conda like an app store for Python packages: you tell it what you want, it figures out what else is needed to make it work, and installs everything together. + +> **Heads up:** GeoLab resets when you sign out, so any packages you installed during a session won't be there next time. The solution is to define your environment in a file (explained below) so you can recreate it anytime. + +--- + +## See What's Already Installed + +Open the **terminal** in GeoLab and try these commands to get your bearings. + +**See all available environments:** + +```bash +conda env list +``` + +The one with a `*` is the currently active environment: + +```text +# conda environments: +# +base * /srv/conda +notebook /srv/conda/envs/notebook +custom_environment /home/jovyan/.conda/envs/custom_environment +``` + +**See all packages in the current environment:** + +```bash +conda list +``` + +This prints a long list. Each row shows a package name, its version, and where it came from: + +```text +# Name Version Build Channel +cartopy 0.24.1 py312h78ddc71_0 conda-forge +numpy 2.2.4 py312h7e3fe57_0 conda-forge +obspy 1.4.1 py312h7b8d3f4_0 conda-forge +xarray 2025.3.0 pyhd8ed1ab_0 conda-forge +… +``` + +--- + +## Installing a Package Without Rebuilding + +If you just need to add one or several packages, you can install them directly from inside a notebook cell using the line magic `%` command. This installs into the notebook's active environment: + +```python +%conda install -c conda-forge pandas +``` + +Or, for packages only available on PyPI (a different package source): + +```python +%pip install earthscope-sdk +``` + +> **Warning:** Installing with `%conda` inside a notebook can use a lot of memory. If your notebook becomes unresponsive or the kernel crashes, use the `environment.yml` approach instead, which is more reliable for larger installs. + +--- + +## Create Your Own Environment + +The best method to create a custom environment is to write a file that lists every package required. Conda reads the file and builds the environment from it. This ensures you can recreate the exact same environment later, and the file can be shared, making the environment reproducible. + +### Step 1: Write an `environment.yml` File + +Open an editor, create a new file called `environment.yml`, and add the following: + +```yml +name: my_environment +channels: + - conda-forge +dependencies: + - python=3.11 + - ipykernel + - numpy + - matplotlib + # Pip-specific packages + - pip: + - seisbench +``` + +Here's what each part means: + +- **name**: What to call the environment. +- **channels**: Where to download packages from (`conda-forge` is a large, reliable source). +- **dependencies**: The packages to be installed. +- `ipykernel` is required so the environment can be used as a notebook kernel, so always include it. +- `- pip:` installs packages only available in the PyPI repository. + +Replace `numpy`, `matplotlib`, or `seisbench` with the required packages. + +### Step 2: Build the Environment + +In the **terminal**, run: + +```bash +conda env create -f environment.yml +``` + +Conda will figure out which versions of everything are compatible and download them. This can take a few minutes, which is normal. + +> **If it fails:** Read the error message. Conda usually names the package that's causing the problem. Try removing it from the file or changing its version. + +### Step 3: Activate the Environment + +```bash +conda activate my_environment +``` + +Activating an environment switches the terminal into that workspace, so any Python commands you run use that environment's packages. The terminal prompt will update to show the environment name, confirming it worked. + +### Step 4: Register It as a Notebook Kernel + +A new environment isn't automatically available in Jupyter notebooks. To use the newly created environment, it has to be registered. Run this command **in the terminal** to register it: + +```bash +python -m ipykernel install --user --name my_environment --display-name "Python (my_environment)" +``` + +### Step 5: Switch to Your Environment in a Notebook + +1. Open a notebook and go to **Kernel > Change Kernel…** + + ![Kernel menu showing the Change Kernel option](../../img/select_kernel.png) + +2. Select **Python (my_environment)**. + + ![Kernel selector dialog with custom environment listed](../../img/select_custom.png) + +3. Check the upper-right corner of the notebook to confirm the kernel changed. + + ![Notebook header showing the active custom environment kernel](../../img/custom_env.png) + +> **Note:** This must be done for each notebook separately. There isn't a way to set it as the default for all notebooks. + +--- + +## Quick Reference for Environments + +| What you want to do | Command | +| --- | --- | +| See all environments | `conda env list` | +| See installed packages | `conda list` | +| Activate an environment | `conda activate ` | +| Leave an environment | `conda deactivate` | +| Build from a file | `conda env create -f environment.yml` | +| Register as a kernel | `python -m ipykernel install --user --name --display-name "