`DataCollection` with large number of events

[qgoestch]

When creating a DataCollection as in the tutorial it takes more and more time to save the DataCollection if the number of events increases.

Here is a minimal working example:

detections = pd.read_csv("detections.csv")
# Create a datacollection with only events from detections
dict = {}
for idx, row in detections.iterrows():
    event_data = da.sel(
        time=slice(row["datetime_start"], row["datetime_stop"]),
        distance=slice(row["distance_start"], row["distance_stop"]),
    )
    dict[f"event_{idx}"] = event_data

dc_detections = xd.DataCollection(dict)
dc_detections.to_netcdf("dc_detections_virtual.nc", virtual=True)

(Tested with 1304 events)

[atrabattoni]

Fixed on feature/datatree-io, will ship in 0.2.9.

The cause was that h5netcdf rescans the whole file on every writable open, and xdas opened the file once per data array — so both saving and opening a collection were quadratic in the number of data arrays. Both directions now use a single shared file handle for the whole collection.

On your case (1304 events), saving went from ~53 min to ~35 s; opening is likewise flat, at about 23 ms per event.

No API change: to_netcdf and open_datacollection behave the same, just linearly.

Thanks for the report and the reproducer.