Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow setting of write_empty_chunks for zarr.core.Array objects #1478

Closed
RKuttruff opened this issue Jul 27, 2023 · 4 comments
Closed

Allow setting of write_empty_chunks for zarr.core.Array objects #1478

RKuttruff opened this issue Jul 27, 2023 · 4 comments

Comments

@RKuttruff
Copy link

When opening a Zarr group via zarr.hierarchy.open_group() there is currently no way to specify write_empty_chunks=False, this can be problematic when appending to existing zarr groups with sparse data as found in pydata/xarray#8009 / pydata/xarray#8016. A simple solution to this would be setting write_empty_chunks=False for Array objects as they are retrieved from the group. Or, perhaps, allowing the user to specify their preference for write_empty_chunks for Arrays in the group when opening/getting the individual Arrays from the group.

@joshmoore
Copy link
Member

cc: @jni

@d-v-b
Copy link
Contributor

d-v-b commented Jul 27, 2023

Thanks @RKuttruff, I have felt this problem acutely.

One fundamental challenge for zarr-python is the friction between a) dynamic options for accessing arrays (like read_only, and write_empty_chunks, among others) and b) the MutableMapping API used by zarr groups to represent their contents. As long as we support array = group['my_array'], it's hard to figure out where a user can provide the dynamic options.

As a stopgap, we should make these dynamic options correspond to set-able properties of arrays, so users can modify the arrays after getting them from the group. I find this really distasteful, but it should work.

@jhamman
Copy link
Member

jhamman commented Jul 27, 2023

I see there being two options for making this work given the current API.

First, lean into using the state of the Array via setters:

array = group['my_array']
array.write_empty_chunks = False
array[:] = ...

Second, utilize some sort of scoped config:

with zarr.config.set(write_empty_chunks=False):
    array = group['my_array']
    array[:] = ...

Neither are perfect but I think I would lean toward the latter option if I had to.

@jhamman
Copy link
Member

jhamman commented Oct 23, 2024

We're sorting this out for 3.0 over in #2409. I'm going to close this in favor of the more recent ticket.

@jhamman jhamman closed this as completed Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants