-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Draft] Non-kerchunk backend for HDF5/netcdf4 files. #87
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking great so far @sharkinsspatial !
kerchunk backend's specialized encoding translation logic
This part I would really like to either factor out, or at a least really understand what it's doing. See #68
virtualizarr/readers/hdf.py
Outdated
@@ -0,0 +1,206 @@ | |||
from typing import List, Mapping, Optional | |||
|
|||
import fsspec |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does one need fsspec if reading a local file? Is there any other way to read from S3 without fsspec at all?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not with a filesystem-like API. You would have to use boto3 or aiobotocore directly.
This is one of the great virtues of fsspec and is not to be under-valued.
virtualizarr/readers/hdf.py
Outdated
def virtual_vars_from_hdf( | ||
path: str, | ||
drop_variables: Optional[List[str]] = None, | ||
) -> Mapping[str, xr.Variable]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this an a way to interface with the code in open_virtual_dataset
This looks cool @sharkinsspatial! My opinion is that it doesn't make sense to just forklift the kerchunk code into virtualizarr. What I would love to see is an extremely tight, strictly typed, unit-tested total refactor of the parsing logic. I think you're headed down the right path, but I encourage you to push as far as you can in that direction. |
for more information, see https://pre-commit.ci
@rabernat Fully agree with your take above 👆 👍 . I'm trying to work through this incrementally whenever I can find some spare time. In the spirit of thorough test coverage 🎊 looking through your issue pydata/xarray#7388 and the corresponding PR I'm not sure what the proper incantation of variable encoding configuration is to use |
for more information, see https://pre-commit.ci
@TomNicholas I'll try to get #261 incorporated in this branch this week. Before tackling this, I'm still dealing with this failing test. I'm still a bit unclear on the mechanics here but I think I may understand. I was previously ignoring empty HDF datasets (which is the same behavior as kerchunk's reader) but to pass the tests introduced in #205 I introduced empty variable support. But as a side effect, this causes a roundtripping failure since these empty variables now get treated as coordinates. This is related to @keewis's recent #260. So I think I have 2 questions
|
I actually don't know, but I would find out by creating a HDF file with an empty variable, opening it with xarray, and seeing if that result is represented the same way as what you have done here. However if the definition of empty variable should just mean "a zarr array with no chunks, just the default Ultimately what matters is that opening HDF via xarray and opening virtual zarr that points to HDF via xarray give the same result.
VirtualiZarr has full control over which variables we choose to make coordinates. I've tried to isolate that logic by in VirtualiZarr/virtualizarr/readers/common.py Line 128 in 4b7612e
coord_names are the names of variables explicitly labeled as COORDINATES in the files' variable-level metadata. Does that help?
You mean the fact they are not FYI that re-implementation of determining coordinates is the reason there is a bug that causes an inconsistency with xarray's behaviour, see #224. |
438b807
to
462aeea
Compare
462aeea
to
1027395
Compare
1027395
to
4362656
Compare
4362656
to
8be9c65
Compare
8be9c65
to
3ab90c6
Compare
127b3d6
to
81874e0
Compare
This is a rudimentary initial implementation for #78. The core code is ported directly from kerchunk's hdf backend. I have not ported the bulk of the kerchunk backend's specialized encoding translation logic but I'll try to do so incrementally so that we can build complete test coverage for the many edge cases it currently covers.