Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
6176f09
Introducing modes
forman Mar 19, 2025
bf6fb99
introduced `mode` arg incl validation
forman Mar 19, 2025
3f6312e
update wrt mode
forman Mar 19, 2025
9d294f4
added Python API
forman Mar 19, 2025
e499f50
renamed `mode` into `op_mode`, op_mode `"native"` works
forman Mar 19, 2025
8106867
Removing common group name prefixes
forman Mar 20, 2025
19089e7
Introduced new module "spatial" and "timeit" for performance testing
forman Mar 21, 2025
61db14f
try supporting CEPH bucket naming
forman Mar 24, 2025
590663f
try supporting CEPH bucket naming (2)
forman Mar 24, 2025
72e19a3
investigating why there is no chunking still
forman Mar 24, 2025
78cbdad
investigating why there is no chunking still
forman Mar 24, 2025
14a69de
remove prints
forman Mar 24, 2025
12310c4
chunking works
forman Mar 25, 2025
c330fa9
analysis mode works for MSI L1C
forman Mar 25, 2025
b9f18d6
analysis mode works for MSI L1C
forman Mar 25, 2025
e51a3fc
sort vars
forman Mar 25, 2025
f67cd30
Added MSIL2A and params
forman Mar 26, 2025
20760ac
fix tests
forman Mar 26, 2025
8bcf5ac
added TODOs and (c)
forman Mar 26, 2025
2aa77df
added CRS to S2
forman Mar 26, 2025
d524eec
refactor out assign_grid_mapping(); more TODOs
forman Mar 27, 2025
126d0be
fixed factors
forman Mar 28, 2025
6112102
generalized resampling
forman Mar 28, 2025
b59a0ef
be less verbose
forman Mar 28, 2025
47eab1b
using new ProdzctType.get_applicable_params()
forman Mar 28, 2025
31e0b70
added missing TODOs
forman Mar 28, 2025
0ca8742
updated docs
forman Mar 28, 2025
366ed86
list all files in bucket
konstntokas Apr 3, 2025
fa7f75a
added "scl" variable with extra attrs
forman Apr 9, 2025
68e9501
fix docstring
forman Apr 9, 2025
08d981c
implementing correct resampling
forman Apr 9, 2025
9a3c39b
added color mapping for SCL
forman Apr 9, 2025
64a6034
Merge branch 'forman-6-introduce_modes' into forman-6-introduce_modes…
forman Apr 9, 2025
c593161
fallback to "min" for SCL
forman Apr 10, 2025
7d01b2d
no numba needed
forman Apr 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Local test data
/xcube-viewer/

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
Expand Down
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,15 @@ pytest
```
By default, this will run all unit and integration tests. To run only the unit test
suite, use:

```shell
pytest tests
```

To run tests and generate a coverage report, use:

```shell
pytest tests/
pytest --cov xarray_eopf --cov-report html tests
```

### Documentation
Expand Down
3 changes: 3 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Backend API Reference

::: xarray_eopf.backend.EopfBackend
108 changes: 108 additions & 0 deletions docs/guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
The xarray backend for EOPF data products `"eopf-zarr"` has two modes of operation,
namely _analysis mode_ and _native mode_, which are described in the following.

## Analysis Mode

This mode aims at representing the EOPF data products in an analysis-ready and
convenient form using the `xarray` data models `DataTree` and `Dataset`.

By default, data products are provided using a common single grid-mapping
for all data variables. That is, spatial up- and downscaling is applied
to selected variables in order to use only a single pair of `x` and `y`
coordinates in the returned datasets.

### Function `open_datatree()`

Synopsis:

```python
datatree = xr.open_datatree(
filename_or_obj,
engine="eopf-zarr",
op_mode="analysis",
**kwargs
)
```

_Not implemented yet._

### Function `open_dataset()`

Synopsis:

```python
dataset = xr.open_dataset(
filename_or_obj,
engine="eopf-zarr",
op_mode="analysis",
**kwargs
)
```

Works basically as `open_datatree()` but using flattened Zarr groups, e.g., groups
`r10m`, `r20m`, `r60m` in Sentinel 2 MSI products.
Flattening includes renaming variables and dimensions by prefixing them using the
concatenated names of nested groups.

Parameters `**params`:

- `resolution`: Target resolution for all spatial data variables / bands.
Must be one of `10`, `20`, or `60`.
- `spline_order`: Spline order to be used for resampling
spatial data variables / bands.
Must be one of `0` (nearest neighbor), `1` (linear), `2` (bi-linear), or
`3` (cubic).
- `variables`: Variables to include in the dataset. Can be a name or regex pattern
or iterable of the latter.
- `product_type_name`: Product type name, such as `"S2B_MSIL1C"`.
Only required if `filename_or_obj` is not a path or URL
that refers to a product path adhering to EOPF naming conventions.


## Native Mode

The aim of this mode is to represent EOPF data products without modification
using the `xarray` data models `DataTree` and `Dataset`. Content and structure
of the original data products are preserved to a maximum extend.

### Function `open_datatree()`

Synopsis:

```python
datatree = xr.open_datatree(
filename_or_obj,
engine="eopf-zarr",
op_mode="native",
**kwargs
)
```

Opens a data product as-is including Zarr groups and returns a data tree object.

### Function `open_dataset()`

Synopsis:

```python
dataset = xr.open_dataset(
filename_or_obj,
engine="eopf-zarr",
op_mode="native",
**kwargs
)
```

Returns a "flattened" version of the data tree returned by `xr.open_datatree()`
in native mode. Groups are removed by turning their contents into individual datasets
and merging them into one. Variables and dimensions are prefixed using their original
group paths to make them unique in the returned dataset. For example, the variable
`b02` found in the group `measurements/reflectance/r10m` will be renamed to
`measurements_reflectance_r10m_b02` using the default underscore group separator.
The separator character is configurable by setting the `group_sep` parameter.

Parameters `**params`:

- `group_sep`: Separator string used to concatenate groups names
to create prefixes for unique variable and dimension names.
Defaults to the underscore character (`"_"`).
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@ object including groups:

The backend supports two distinct modes of operation, `"native"` and `"analysis-ready"`:

- `mode="native"` - using the _native mode_, the returned `Dataset` or `DataTree`
- `op_mode="native"` - using the _native mode_, the returned `Dataset` or `DataTree`
objects try to serve as a 1:1 representation of the actual Zarr product structure and
content with either none or minimal preprocessing applied.

- `mode="analysis"` - using the _analysis mode_, the returned `Dataset`
- `op_mode="analysis"` - using the _analysis mode_, the returned `Dataset`
or `DataTree` objects attempt to be user-friendly and analysis-ready to a maximum
extend. Certain preprocessing steps will be applied depending on the specific
Sentinel product towards an analysis-ready data model and content. For example,
Expand Down
3 changes: 3 additions & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,12 @@ dependencies:
# Library Dependencies
- aiohttp
- dask
- dask-image
- fsspec
- numpy
- pyproj
- requests
- s3fs
- xarray >=2024.10
- zarr >=2,<3.0
# Development Dependencies - Tools
Expand Down
Loading