Skip to content

writing to netcdf fails with newer dependency versions #190

Description

@veenstrajelmer

Writing a netcdf file from a xarray Dataset generated with ddlpy.ddlpy.dataframe_to_xarray() raises `"TypeError: Cannot interpret '<StringDtype(na_value=nan)>' as a data type"`` with pandas 3 (or old xarray version) in dfm_tools. To reproduce:

import ddlpy
import datetime as dt

locations = ddlpy.locations()

bool_grootheid = locations["Grootheid.Code"] == "WATHTE"
bool_groepering = locations["Groepering.Code"] == ""
bool_procestype = locations["ProcesType"] == "meting"
location = locations[bool_grootheid & bool_groepering & bool_procestype].loc[
    "denhelder.marsdiep"
]

start_date = dt.datetime(1953, 1, 1)
end_date = dt.datetime(1953, 4, 1)
measurements = ddlpy.measurements(
    location, start_date=start_date, end_date=end_date
)

always_preserve = [
    "WaarnemingMetadata.Statuswaarde",
    "WaarnemingMetadata.Kwaliteitswaardecode",
]
ds_clean = ddlpy.dataframe_to_xarray(
    df=measurements,
    always_preserve=always_preserve,
)

ds_clean.to_netcdf(r"c:\Users\veenstra\Downloads\dfmt_envs\test_ddlpy.nc", format="NETCDF4_CLASSIC")

Todo:

  • add to tests, writing to netcdf is currently not covered
  • indeed caused by pandas 3? Might also be numpy/xarray versions >> cannot reproduce outside dfm_tools_env
  • fix it by converting strings to numpy arrays on forehand >> cannot reproduce
  • also might prevent the need to use format="NETCDF4_CLASSIC"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions