Skip to content

pixi solve strategy lowest and highest#1485

Merged
deltamarnix merged 32 commits into
mainfrom
feat/pixi-solve-strategy-lowest-4
Jun 12, 2026
Merged

pixi solve strategy lowest and highest#1485
deltamarnix merged 32 commits into
mainfrom
feat/pixi-solve-strategy-lowest-4

Conversation

@LuukBlom

@LuukBlom LuukBlom commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator

Explanation

Add lowest dependency solving to the test matrix.
Only use pypi, and no conda packages anymore. This works because we removed GDAL and instead use rio-vrt as dependency.

General Checklist

  • Updated tests or added new tests
  • Branch is up to date with main
  • Tests & pre-commit hooks pass
  • Updated documentation
  • Updated changelog.rst

Data/Catalog checklist

  • data/catalogs/predefined_catalogs.yml has not been modified.
  • None of the old data_catalog.yml files have been changed
  • data/changelog.rst has been updated
  • new file uses LF line endings (done automatically if you used update_versions.py)
  • New file has been tested locally
  • Tests have been added using the new file in the test suite

Additional Notes (optional)

Add any additional notes or information that may be helpful.

LuukBlom and others added 21 commits May 12, 2026 16:51
add minimum versions to all dependencies.
add low pixi envs to the ci testing
- add back type hints
- fix asserts
…_vrt to replace osgeo.BuildVRT(). add/update tests
…d _build_vrt to replace osgeo.BuildVRT(). add/update tests"

This reverts commit c43bce4.
And pins numpy in conda-forge dependencies.
This way we have more conda-forge dependencies, but it's the only transient
dependency of gdal. So adding this will ensure that the numpy version
is similar to the one available in pypi. It's still no guarantee.

# Conflicts:
#	pixi.lock
#	pyproject.toml
Comment thread .github/workflows/docs.yml Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates HydroMT’s dependency management and CI to support “lowest-direct” dependency solving via pixi, while removing the explicit GDAL Python bindings requirement by switching VRT tile caching to rio-vrt. It also adjusts tests and docs to match the new dependency approach (PyPI-only focus) and updated tooling.

Changes:

  • Add a new pixi “lowest-deps” solve strategy and run it in the GitHub Actions test matrix.
  • Remove HAS_GDAL compatibility plumbing and switch VRT caching from osgeo.gdal.BuildVRT to rio_vrt.build_vrt, updating tests accordingly.
  • Refresh CI/docs tooling (pixi version bump, pandoc installation guidance and workflow step) and modernize some typing / test assertions.

Reviewed changes

Copilot reviewed 23 out of 24 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tests/test_plugins.py Adjusts entry point mocking to only intercept the components entry point group.
tests/data_catalog/test_data_catalog.py Removes GDAL-based skip conditions from integration tests now that GDAL bindings are no longer required.
tests/data_catalog/drivers/raster/test_rasterio_driver.py Removes GDAL skip/import to run raster caching tests unconditionally.
tests/data_catalog/drivers/raster/test_raster_xarray_driver.py Makes coord assertions order-independent.
tests/data_catalog/drivers/geodataset/test_xarray_driver.py Makes coord assertions order-independent.
tests/data_catalog/drivers/dataset/test_dataset_xarray_driver.py Makes coord assertions order-independent.
tests/conftest.py Moves global test configuration into session autouse fixtures and guards xarray option by version.
tests/_utils/test_caching.py Removes GDAL-dependent skips and a GDAL-missing error-path test.
pyproject.toml Updates dependency lower bounds, adds rio-vrt, adds “lowest-deps” pixi feature, updates pixi environments/CI targets.
hydromt/data_catalog/uri_resolvers/uri_resolver.py Modernizes type hints to `X
hydromt/data_catalog/sources/data_source.py Modernizes typing (dict[str, …], list[str]) in a few methods.
hydromt/data_catalog/data_catalog.py Removes outdated docstring note about requiring GDAL Python bindings for caching.
hydromt/_utils/caching.py Replaces GDAL BuildVRT usage with rio_vrt.build_vrt and removes GDAL import/guard.
hydromt/_compat.py Removes HAS_GDAL detection flag.
docs/dev/plugin_dev/example_test.yml Bumps pixi version used in the example workflow.
docs/dev/core_dev/dev_install.rst Documents pandoc as an external (pixi-global) requirement for docs builds.
.github/workflows/tests.yml Adds “low” dependency environment to test matrix and bumps pixi version.
.github/workflows/sonar.yml Bumps pixi version.
.github/workflows/pixi_auto_update.yml Bumps pixi version.
.github/workflows/downstream-plugins.yml Bumps pixi version.
.github/workflows/docs.yml Bumps pixi version and installs pandoc via pixi global install.
.github/workflows/create-release.yml Bumps pixi version.
.github/workflows/check-data-catalogs.yml Bumps pixi version.
Comments suppressed due to low confidence (1)

hydromt/data_catalog/sources/data_source.py:174

  • _get_relative_uri is annotated to always return Path, but it can return None when handle_nodata is set to ignore (see the return None branch). Update the return type to Path | None to reflect actual behavior and avoid type/IDE confusion.
        return res

    def _get_relative_uri(self, handle_nodata: NoDataStrategy, **query_kwargs) -> Path:
        uri = PurePath(self.uri)
        if "{" in self.uri:
            # first resolve any placeholders
            uris: list[str] = self.uri_resolver.resolve(
                uri=self.full_uri,
                handle_nodata=handle_nodata,
                **query_kwargs,
            )
            if not uris:
                return None  # handle_nodata == ignore
            # if multiple_uris, use the first one:

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pyproject.toml
Comment thread pyproject.toml
Comment thread pyproject.toml
Comment thread hydromt/data_catalog/uri_resolvers/uri_resolver.py Outdated
LuukBlom and others added 3 commits June 8, 2026 12:09
@LuukBlom

LuukBlom commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator Author

rio-vrt has only 1 maintainer, and the last commit and release was done 2 years ago.

I had to introduce a workaround for our tests since rio-vrt.build_vrt has a bug, resulting in TypeErrors when the tiles list has only 1 element.
I dont think this will happen often outside of tests, but its a cryptic error that we should handle imo.

There are 2 things we an easily do:

  1. have this workaround solution that deduplicates the xml (this passes our tests)
    # Pass duplicated list, then strip the duplicate source from the written XML.
    duplicated = False
    if len(new) == 1:
        new = new + new
        duplicated = True

    build_vrt(vrt_destination_path, new)

    # de-duplicate
    if duplicated:
        tree = ET.parse(vrt_destination_path)
        for band in tree.getroot().findall("VRTRasterBand"):
            source_els = [s for s in band if s.tag.endswith("Source")]
            if len(source_els) > 1:
                band.remove(source_els[-1])
        tree.write(vrt_destination_path, xml_declaration=True, encoding="UTF-8")

OR

  1. raise an informative error when there is only 0 or 1 files.
    if len(new) <= 1:
        raise ValueError("Need at least 2 files when creating a vrt")
    build_vrt(vrt_destination_path, new)

1 is a bit ugly, but 2 breaks usage when users query a small bbox/region that only intersects 1 tile.

Any opinions @deltamarnix ?

@LuukBlom

LuukBlom commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator Author

I created an issue for the bug in the rio-vrt repo: 12rambau/rio-vrt#26

It seems to not be maintained actively anymore, so good to manage expectations.

Comment thread hydromt/_utils/caching.py
@deltamarnix deltamarnix merged commit dbed3fc into main Jun 12, 2026
59 of 60 checks passed
@deltamarnix deltamarnix deleted the feat/pixi-solve-strategy-lowest-4 branch June 12, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants