Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
6176f09
Introducing modes
forman Mar 19, 2025
bf6fb99
introduced `mode` arg incl validation
forman Mar 19, 2025
3f6312e
update wrt mode
forman Mar 19, 2025
9d294f4
added Python API
forman Mar 19, 2025
e499f50
renamed `mode` into `op_mode`, op_mode `"native"` works
forman Mar 19, 2025
8106867
Removing common group name prefixes
forman Mar 20, 2025
19089e7
Introduced new module "spatial" and "timeit" for performance testing
forman Mar 21, 2025
61db14f
try supporting CEPH bucket naming
forman Mar 24, 2025
590663f
try supporting CEPH bucket naming (2)
forman Mar 24, 2025
72e19a3
investigating why there is no chunking still
forman Mar 24, 2025
78cbdad
investigating why there is no chunking still
forman Mar 24, 2025
14a69de
remove prints
forman Mar 24, 2025
12310c4
chunking works
forman Mar 25, 2025
c330fa9
analysis mode works for MSI L1C
forman Mar 25, 2025
b9f18d6
analysis mode works for MSI L1C
forman Mar 25, 2025
e51a3fc
sort vars
forman Mar 25, 2025
f67cd30
Added MSIL2A and params
forman Mar 26, 2025
20760ac
fix tests
forman Mar 26, 2025
8bcf5ac
added TODOs and (c)
forman Mar 26, 2025
2aa77df
added CRS to S2
forman Mar 26, 2025
d524eec
refactor out assign_grid_mapping(); more TODOs
forman Mar 27, 2025
126d0be
fixed factors
forman Mar 28, 2025
6112102
generalized resampling
forman Mar 28, 2025
b59a0ef
be less verbose
forman Mar 28, 2025
47eab1b
using new ProdzctType.get_applicable_params()
forman Mar 28, 2025
31e0b70
added missing TODOs
forman Mar 28, 2025
0ca8742
updated docs
forman Mar 28, 2025
366ed86
list all files in bucket
konstntokas Apr 3, 2025
fa7f75a
added "scl" variable with extra attrs
forman Apr 9, 2025
68e9501
fix docstring
forman Apr 9, 2025
08d981c
implementing correct resampling
forman Apr 9, 2025
9a3c39b
added color mapping for SCL
forman Apr 9, 2025
64a6034
Merge branch 'forman-6-introduce_modes' into forman-6-introduce_modes…
forman Apr 9, 2025
c593161
fallback to "min" for SCL
forman Apr 10, 2025
7d01b2d
no numba needed
forman Apr 10, 2025
8ba10af
Merge pull request #37 from EOPF-Sample-Service/forman-6-introduce_mo…
forman Apr 10, 2025
1eb1311
renamed ProductType --> AnalysisMode
forman Apr 10, 2025
c580089
added tests for amodes
forman Apr 10, 2025
8f292dd
done utils tests
forman Apr 10, 2025
b063cf4
done amode tests
forman Apr 10, 2025
ef81080
done spatial tests
forman Apr 10, 2025
5e40304
done backend tests
forman Apr 10, 2025
0773c6b
added a couple of sentinel2 tests
forman Apr 10, 2025
da7e28f
creating more realistic sentinel-2 dummies
forman Apr 10, 2025
6fd03b6
moved test helpers into dedicated subpackage
forman Apr 11, 2025
b551387
sentinel2 tests complete
forman Apr 11, 2025
e83526e
store tests complete
forman Apr 11, 2025
2e6be27
docstrings
forman Apr 11, 2025
68bbe4c
improved backend tests
forman Apr 11, 2025
132d243
Update docs/guide.md
forman Apr 14, 2025
485eecb
Update xarray_eopf/flatten.py
forman Apr 14, 2025
61f1350
updated docs after PR review
forman Apr 14, 2025
67fbd6f
added history attribute to rescaled variables
forman Apr 14, 2025
dbcdc36
removed print()
forman Apr 14, 2025
5aabd76
removed unused constants
forman Apr 14, 2025
3615d4b
run isort/ruff
forman Apr 14, 2025
756bba5
update
forman Apr 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Local test data
/xcube-viewer/

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
Expand Down
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,15 @@ pytest
```
By default, this will run all unit and integration tests. To run only the unit test
suite, use:

```shell
pytest tests
```

To run tests and generate a coverage report, use:

```shell
pytest tests/
pytest --cov xarray_eopf --cov-report html tests
```

### Documentation
Expand Down
3 changes: 3 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Backend API Reference

::: xarray_eopf.backend.EopfBackend
117 changes: 117 additions & 0 deletions docs/guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
The xarray backend for EOPF data products `"eopf-zarr"` has two modes of operation,
namely _analysis mode_ (the default) and _native mode_, which are described in
the following.

## Analysis Mode

This mode aims at representing the EOPF data products in an analysis-ready and
convenient form using the `xarray` data models `DataTree` and `Dataset`.
For this reason, it is the default mode of operation when using the `"eopf-zarr"`
backend.

The data products are provided in this mode use a unified grid mapping
for all their data variables. This means that selected variables are
spatially up-scaled or down-scaled as needed, so that the dataset can use a
single shared pair of `x` and `y` coordinates in the returned datasets.

### Function `open_dataset()`

Synopsis:

```python
dataset = xr.open_dataset(
filename_or_obj,
engine="eopf-zarr",
op_mode="analysis",
**kwargs
)
```

Returns a EOPF data product from Sentinel-1, -2, or -3 in a analysis-ready, convenient
form. All bands and quality flags are resampled to a unified, user-provided resolution.

Parameters `**params`:

- `resolution`: Target resolution for all spatial data variables / bands.
Must be one of `10`, `20`, or `60`.
- `spline_order`: Spline order to be used for resampling
spatial data variables / bands.
Must be one of `0` (nearest neighbor), `1` (linear), `2` (bi-linear), or
`3` (cubic).
- `variables`: Variables to include in the dataset. Can be a name or regex pattern
or iterable of the latter.
- `product_type`: Product type name, such as `"S2B_MSIL1C"`.
Only required if `filename_or_obj` is not a path or URL
that refers to a product path adhering to EOPF naming conventions.


### Function `open_datatree()`

Synopsis:

```python
datatree = xr.open_datatree(
filename_or_obj,
engine="eopf-zarr",
op_mode="analysis",
**kwargs
)
```

This function is currently not implemented for the analysis mode
and will raise a `NotImplementedError`.

## Native Mode

The aim of this mode is to represent EOPF data products without modification
using the `xarray` data models `DataTree` and `Dataset`. Content and structure
of the original data products are preserved to a maximum extend.

### Function `open_datatree()`

Synopsis:

```python
datatree = xr.open_datatree(
filename_or_obj,
engine="eopf-zarr",
op_mode="native",
**kwargs
)
```

Opens a data product as-is including Zarr groups and returns a data tree object.

This function currently returns the result of calling
`xr.open_datatree(filename_or_obj, engine="zarr", **kwargs)`.

### Function `open_dataset()`

Synopsis:

```python
dataset = xr.open_dataset(
filename_or_obj,
engine="eopf-zarr",
op_mode="native",
**kwargs
)
```

Returns a "flattened" version of the data tree returned by `xr.open_datatree()`
in native mode. Groups are removed by turning their contents into individual datasets
and merging them into one. Variables and dimensions are prefixed using their original
group paths to make them unique in the returned dataset. For example, the variable
`b02` found in the group `measurements/reflectance/r10m` will be renamed to
`measurements_reflectance_r10m_b02` using the default underscore group separator.
The separator character is configurable by setting the `group_sep` parameter.

The main use case for this function is to allow passing an EOPF data product
where the type `xr.Dataset` is expected (not `xr.DataTree`) and where the naming of
dimensions and variables is not an issue.

Parameters `**params`:

- `group_sep`: Separator string used to concatenate groups names
to create prefixes for unique variable and dimension names.
Defaults to the underscore character (`"_"`).
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@ object including groups:

The backend supports two distinct modes of operation, `"native"` and `"analysis-ready"`:

- `mode="native"` - using the _native mode_, the returned `Dataset` or `DataTree`
- `op_mode="native"` - using the _native mode_, the returned `Dataset` or `DataTree`
objects try to serve as a 1:1 representation of the actual Zarr product structure and
content with either none or minimal preprocessing applied.

- `mode="analysis"` - using the _analysis mode_, the returned `Dataset`
- `op_mode="analysis"` - using the _analysis mode_, the returned `Dataset`
or `DataTree` objects attempt to be user-friendly and analysis-ready to a maximum
extend. Certain preprocessing steps will be applied depending on the specific
Sentinel product towards an analysis-ready data model and content. For example,
Expand Down
3 changes: 3 additions & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,12 @@ dependencies:
# Library Dependencies
- aiohttp
- dask
- dask-image
- fsspec
- numpy
- pyproj
- requests
- s3fs
- xarray >=2024.10
- zarr >=2,<3.0
# Development Dependencies - Tools
Expand Down
Loading