Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function to combine Kerchunks into datatree kerchunks. #12

Open
abkfenris opened this issue Oct 12, 2023 · 0 comments
Open

Function to combine Kerchunks into datatree kerchunks. #12

abkfenris opened this issue Oct 12, 2023 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@abkfenris
Copy link
Owner

abkfenris commented Oct 12, 2023

🚀 Feature Request

Add a function that can take a dictionary of references to Kerchunk files and combine them into a single datatree kerchunk with a similar interface to

def from_dict(
datasets: Dict["DatetimeScalar", xr.Dataset],
group_prefix: str = constants.DEFAULT_GROUP_PREFIX,
time_format: str = constants.DEFAULT_TIME_FORMAT,
time_coord: Optional[str] = None,
forecast_coords: Optional[List[str]] = None,
tree_attrs: Optional[dict[str, Any]] = None,
) -> datatree.DataTree:
"""Build a datatree from a dictionary of forecast reference times to datasets"""
path_dict = {}
for dt, ds in datasets.items():
path = model_run_path(dt, group_prefix, time_format)
path_dict[path] = ds
tree = datatree.DataTree.from_dict(path_dict)
if tree_attrs is None:
tree_attrs = {}
tree_attrs[constants.DT_ATTR_GROUP_PREFIX] = group_prefix
tree_attrs[constants.DT_ATTR_TIME_FORMAT] = time_format
if time_coord:
tree_attrs[constants.DT_ATTR_TIME_COORD] = time_coord
if forecast_coords:
tree_attrs[constants.DT_ATTR_FORECAST_COORDS] = forecast_coords
tree.attrs = tree_attrs
return tree

🔈 Motivation

It would be really nice to be able to be more lazy when building datatrees.

🛰 Alternatives

Creative Kerchunking ahead of time into each variety of dataset similar to what Matt is doing.

📎 Additional context

See fsspec/kerchunk#357 for example on how to build up the json.

Kerchunk reference spec

@abkfenris abkfenris added the enhancement New feature or request label Oct 12, 2023
@abkfenris abkfenris self-assigned this Oct 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant