Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow writing DataTree objects containing references to disk #244

Open
keewis opened this issue Oct 2, 2024 · 1 comment
Open

allow writing DataTree objects containing references to disk #244

keewis opened this issue Oct 2, 2024 · 1 comment
Labels
enhancement New feature or request xarray Requires changes to xarray upstream

Comments

@keewis
Copy link
Contributor

keewis commented Oct 2, 2024

In trying to create a nicer way to access ocean model output (several stacks of netcdf files where each stack can be concatenated, but not necessarily merged into a single Dataset object), I've been able to construct a DataTree object:

hourly = xr.concat([virtualizarr.open_virtual_dataset(path, ...) for path in paths], dim="time", ...)
daily = xr.concat([virtualizarr.open_virtual_dataset(path, ...) for path in paths], dim="time", ...)
monthly = xr.concat([virtualizarr.open_virtual_dataset(path, ...) for path in paths], dim="time", ...)

tree = DataTree.from_dict({"/": ..., "/hourly": hourly, "/daily": daily, "/monthly": monthly})

but would then need a way to write that tree to disk.

The current file formats (except maybe parquet, but not sure), definitely support this since they're based on zarr, we'd just need to create a DataTree accessor and write the code to serialize DataTree objects containing ManifestArrays.

Edit: related to #84 and #11

@TomNicholas TomNicholas added enhancement New feature or request xarray Requires changes to xarray upstream labels Oct 2, 2024
@TomNicholas
Copy link
Member

I was going to say that this is a duplicate of #84, but it's actually not because being able to write from DataTree is useful even if we have not yet implemented open_virtual_datatree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request xarray Requires changes to xarray upstream
Projects
None yet
Development

No branches or pull requests

2 participants