pixi solve strategy lowest and highest#1485
Conversation
add minimum versions to all dependencies. add low pixi envs to the ci testing
- add back type hints - fix asserts
…ed to be installed by the user.
…_vrt to replace osgeo.BuildVRT(). add/update tests
…d _build_vrt to replace osgeo.BuildVRT(). add/update tests" This reverts commit c43bce4. And pins numpy in conda-forge dependencies. This way we have more conda-forge dependencies, but it's the only transient dependency of gdal. So adding this will ensure that the numpy version is similar to the one available in pypi. It's still no guarantee. # Conflicts: # pixi.lock # pyproject.toml
There was a problem hiding this comment.
Pull request overview
This PR updates HydroMT’s dependency management and CI to support “lowest-direct” dependency solving via pixi, while removing the explicit GDAL Python bindings requirement by switching VRT tile caching to rio-vrt. It also adjusts tests and docs to match the new dependency approach (PyPI-only focus) and updated tooling.
Changes:
- Add a new pixi “lowest-deps” solve strategy and run it in the GitHub Actions test matrix.
- Remove
HAS_GDALcompatibility plumbing and switch VRT caching fromosgeo.gdal.BuildVRTtorio_vrt.build_vrt, updating tests accordingly. - Refresh CI/docs tooling (pixi version bump, pandoc installation guidance and workflow step) and modernize some typing / test assertions.
Reviewed changes
Copilot reviewed 23 out of 24 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_plugins.py |
Adjusts entry point mocking to only intercept the components entry point group. |
tests/data_catalog/test_data_catalog.py |
Removes GDAL-based skip conditions from integration tests now that GDAL bindings are no longer required. |
tests/data_catalog/drivers/raster/test_rasterio_driver.py |
Removes GDAL skip/import to run raster caching tests unconditionally. |
tests/data_catalog/drivers/raster/test_raster_xarray_driver.py |
Makes coord assertions order-independent. |
tests/data_catalog/drivers/geodataset/test_xarray_driver.py |
Makes coord assertions order-independent. |
tests/data_catalog/drivers/dataset/test_dataset_xarray_driver.py |
Makes coord assertions order-independent. |
tests/conftest.py |
Moves global test configuration into session autouse fixtures and guards xarray option by version. |
tests/_utils/test_caching.py |
Removes GDAL-dependent skips and a GDAL-missing error-path test. |
pyproject.toml |
Updates dependency lower bounds, adds rio-vrt, adds “lowest-deps” pixi feature, updates pixi environments/CI targets. |
hydromt/data_catalog/uri_resolvers/uri_resolver.py |
Modernizes type hints to `X |
hydromt/data_catalog/sources/data_source.py |
Modernizes typing (dict[str, …], list[str]) in a few methods. |
hydromt/data_catalog/data_catalog.py |
Removes outdated docstring note about requiring GDAL Python bindings for caching. |
hydromt/_utils/caching.py |
Replaces GDAL BuildVRT usage with rio_vrt.build_vrt and removes GDAL import/guard. |
hydromt/_compat.py |
Removes HAS_GDAL detection flag. |
docs/dev/plugin_dev/example_test.yml |
Bumps pixi version used in the example workflow. |
docs/dev/core_dev/dev_install.rst |
Documents pandoc as an external (pixi-global) requirement for docs builds. |
.github/workflows/tests.yml |
Adds “low” dependency environment to test matrix and bumps pixi version. |
.github/workflows/sonar.yml |
Bumps pixi version. |
.github/workflows/pixi_auto_update.yml |
Bumps pixi version. |
.github/workflows/downstream-plugins.yml |
Bumps pixi version. |
.github/workflows/docs.yml |
Bumps pixi version and installs pandoc via pixi global install. |
.github/workflows/create-release.yml |
Bumps pixi version. |
.github/workflows/check-data-catalogs.yml |
Bumps pixi version. |
Comments suppressed due to low confidence (1)
hydromt/data_catalog/sources/data_source.py:174
_get_relative_uriis annotated to always returnPath, but it can returnNonewhenhandle_nodatais set to ignore (see thereturn Nonebranch). Update the return type toPath | Noneto reflect actual behavior and avoid type/IDE confusion.
return res
def _get_relative_uri(self, handle_nodata: NoDataStrategy, **query_kwargs) -> Path:
uri = PurePath(self.uri)
if "{" in self.uri:
# first resolve any placeholders
uris: list[str] = self.uri_resolver.resolve(
uri=self.full_uri,
handle_nodata=handle_nodata,
**query_kwargs,
)
if not uris:
return None # handle_nodata == ignore
# if multiple_uris, use the first one:
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
I had to introduce a workaround for our tests since There are 2 things we an easily do:
# Pass duplicated list, then strip the duplicate source from the written XML.
duplicated = False
if len(new) == 1:
new = new + new
duplicated = True
build_vrt(vrt_destination_path, new)
# de-duplicate
if duplicated:
tree = ET.parse(vrt_destination_path)
for band in tree.getroot().findall("VRTRasterBand"):
source_els = [s for s in band if s.tag.endswith("Source")]
if len(source_els) > 1:
band.remove(source_els[-1])
tree.write(vrt_destination_path, xml_declaration=True, encoding="UTF-8")OR
if len(new) <= 1:
raise ValueError("Need at least 2 files when creating a vrt")
build_vrt(vrt_destination_path, new)1 is a bit ugly, but 2 breaks usage when users query a small bbox/region that only intersects 1 tile. Any opinions @deltamarnix ? |
|
I created an issue for the bug in the rio-vrt repo: 12rambau/rio-vrt#26 It seems to not be maintained actively anymore, so good to manage expectations. |
Explanation
Add lowest dependency solving to the test matrix.
Only use pypi, and no conda packages anymore. This works because we removed GDAL and instead use rio-vrt as dependency.
General Checklist
mainData/Catalog checklist
data/catalogs/predefined_catalogs.ymlhas not been modified.data_catalog.ymlfiles have been changeddata/changelog.rsthas been updatedLFline endings (done automatically if you usedupdate_versions.py)Additional Notes (optional)
Add any additional notes or information that may be helpful.