Zarr : Allow setting `write_empty_chunks` #8016

RKuttruff · 2023-07-24T21:07:38Z

Closes In Dataset.to_zarr cannot specify to not write empty chunks when appending to existing store #8009

Zarr has an attribute in zarr.core.Array that specifies the behavior desired in this issue. Added a parameter in Dataset.to_zarr, write_empty_chunks to allow the user to explicitly set this behavior.

Tests added

Test case to ensure empty chunks are/are not written on append given the value of write_empty_chunks.

User visible changes (including notable bug fixes) are documented in whats-new.rst

- Incorrectly set default values - Need to set write_empty_chunks for new variables in group

welcome · 2023-07-24T21:07:39Z

Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient.
If you have questions, some answers may be found in our contributing guidelines.

xarray/tests/test_backends.py

Co-authored-by: Deepak Cherian <[email protected]>

for more information, see https://pre-commit.ci

RKuttruff · 2023-07-25T06:44:39Z

Looks like this will require zarr>=2.11 to have an effect; otherwise it will issue a warning of unrecognized kwarg on to_zarr. This seems to be what is causing the min_all_deps check to fail (+ the fact that the new test cases expect it to work when it shouldn't with that version of zarr). Is this something that would be better resolved by a check or bumping the minimum required zarr version?

Illviljan · 2023-07-25T08:02:36Z

If I understand the version policy CI correctly, I think you can bump zarr to 2.12.

xarray/backends/zarr.py

- Default value of None which will fall back to encoding specified behavior or Zarr defaults - If param (!= None) and encoding setting disagree, a ValueError is raised - Test case checks for compatible zarr version - Documented minimum required Zarr version

Illviljan · 2023-07-25T21:51:29Z

Could you change the minimum requirement of zarr to 2.12 as well? It's here:

xarray/ci/requirements/min-all-deps.yml

Line 53 in bb501ba

- zarr=2.10

If the Minimum Version Policy (min-all-deps) CI passes, you can simplify the test and add that change to the what's new. Here's a template you can use for that:

xarray/doc/whats-new.rst

Lines 537 to 558 in bb501ba

    
           Breaking changes 
        
           ~~~~~~~~~~~~~~~~ 
        
           - The minimum versions of some dependencies were changed (:pull:`7300`): 
        
             ========================== =========  ======== 
        
              Package                         Old      New 
        
             ========================== =========  ======== 
        
              boto                           1.18     1.20 
        
              cartopy                        0.19     0.20 
        
              distributed                 2021.09  2021.11 
        
              dask                        2021.09  2021.11 
        
              h5py                            3.1      3.6 
        
              hdf5                           1.10     1.12 
        
              matplotlib-base                 3.4      3.5 
        
              nc-time-axis                    1.3      1.4 
        
              netcdf4                       1.5.3    1.5.7 
        
              packaging                      20.3     21.3 
        
              pint                           0.17     0.18 
        
              pseudonetcdf                    3.1      3.2 
        
              typing_extensions              3.10      4.0 
        
             ========================== =========  ========

Zarr minimum dependency bump should make the version check no longer necessary

RKuttruff · 2023-07-25T22:01:29Z

@Illviljan Done. Bumped zarr version + documented it in whats-new.rst

dcherian · 2023-07-25T22:29:32Z

xarray/tests/test_backends.py

+                store,
+                mode="w",
+                encoding=encoding,
+                write_empty_chunks=write_empty,


Does it make sense to allow this to be controlled on a per-variable basis? If so, encoding would be the right place to specify it. I don't have an opinion, just a question

It's an option for Zarr array creation so it would originally be specified on a per-variable basis; however, the issue is that there is no way to specify it when appending data to an existing store, as there is an error raised when encoding is not empty.

What this reveals is that our usage of encoding is overloaded. There are evidently two distinct types of information that we can put in encoding:

Specifications for how to store the data on disk, e.g. dtype, compression, etc, that must be consistent for all subsequent writes to the Zarr array

Runtime choices, like write_empty_chunks, that can be different for each write (and that don't necessarily need to be known to read the data back)

Perhaps not necessary for this PR, but perhaps we could consider explicitly distinguishing these two distinct types of encoding. Maybe one solution is to eventually remove write_empty_chunks and all such runtime choices from encoding and have them only be supported as kwargs.

@rabernat That sounds like it would be a good idea. I'm sure there are probably more cases where this lack of distinction could cause a problem.

Illviljan · 2023-07-25T22:11:15Z

xarray/backends/zarr.py

@@ -666,6 +671,8 @@ def set_variables(self, variables, check_encoding_set, writer, unlimited_dims=No
                # metadata. This would need some case work properly with region
                # and append_dim.
                zarr_array = self.zarr_group[name]
+                if self._write_empty is not None:
+                    zarr_array._write_empty_chunks = self._write_empty


Looks like a non-public attribute? There's no public alternative?

Not that I could find. There's a way to specify it when opening direct arrays but since xarray goes through zarr.hierarchy.open_group (and there is no equivalent parameter there compared to zarr.creation.open_array, nor kwargs to pass) there is no way to do so, at least from within xarray.

This makes me nervous. How important is it that we support this feature on existing arrays? Alternatively, we can work on making the write_empty_chunks property in Zarr public.

Another option is to change how you create the array. Instead of using zarr_array = self.zarr_group[name], you can create the array directly, e.g.

zarr_array = zarr.open(store=self.zarr_group.store, path=f'{self.zarr_group.name}/{name}', write_empty_chunks=...)

This is longer, but you have complete control over the instantiation of the zarr array

I'll give that a try, may be a good solution pending zarr-developers/zarr-python#1478

xarray/backends/zarr.py

xarray/tests/test_backends.py

xarray/backends/zarr.py

Co-authored-by: Illviljan <[email protected]>

jhamman

A few comments that hopefully help move this forward. Can you open an issue with Zarr to make Array.write_empty_chunks a public setter?

ci/requirements/min-all-deps.yml

doc/whats-new.rst

jhamman · 2023-07-27T04:39:57Z

xarray/backends/zarr.py

@@ -666,6 +671,8 @@ def set_variables(self, variables, check_encoding_set, writer, unlimited_dims=No
                # metadata. This would need some case work properly with region
                # and append_dim.
                zarr_array = self.zarr_group[name]
+                if self._write_empty is not None:
+                    zarr_array._write_empty_chunks = self._write_empty


This makes me nervous. How important is it that we support this feature on existing arrays? Alternatively, we can work on making the write_empty_chunks property in Zarr public.

xarray/core/dataset.py

Co-authored-by: Joe Hamman <[email protected]>

RKuttruff · 2023-07-27T14:32:27Z

@jhamman zarr-developers/zarr-python#1478

RKuttruff · 2023-07-27T14:38:52Z

In response to #8016 (comment)

How important is it that we support this feature on existing arrays?

The issue is that the initial preference for an array is not kept when appending data to that array. In cases where the data is very sparse, this can lead to many unnecessary empty chunks being written which can have all sorts of implications. (#8009). Zarr appears to set this behavior when opening/creating arrays, but not when opening groups (which is how it's handled in the xarray backend), and it is defaulted to True. This change would give the user the choice as they would have if there were manually writing/appending to the arrays, rather than going through to_zarr.

…mpty-chunks

RKuttruff · 2023-07-27T20:03:28Z

Thanks to @d-v-b's suggestion, we're no longer writing to a non-public attribute of the Array object

dcherian · 2023-08-03T20:34:41Z

Thanks @RKuttruff @jhamman can we merge?

welcome · 2023-08-04T15:15:28Z

Congratulations on completing your first pull request! Welcome to Xarray! We are proud of you, and hope to see you again!

dcherian · 2023-08-04T15:15:42Z

Thanks @RKuttruff ! Welcome to Xarray!

RKuttruff · 2023-08-04T15:16:33Z

Thank you so much @dcherian!

- Installs branch from pydata/xarray#8016 - Replaces `{var: {'write_empty_chunks': False}}` with `write_empty_chunks=False` kwarg in `Dataset.to_zarr` calls in `ZarrWriter`

rileykk added 2 commits July 24, 2023 12:39

Added parameter write_empty_chunks to Dataset.to_zarr

c0c0373

Missed a couple things...

010af6d

- Incorrectly set default values - Need to set write_empty_chunks for new variables in group

github-actions bot added topic-backends topic-zarr Related to zarr storage library io labels Jul 24, 2023

dcherian changed the title ~~Write empty chunks~~ Zarr : Allow setting write_empty_chunks Jul 24, 2023

dcherian requested review from rabernat and jhamman July 24, 2023 21:16

dcherian reviewed Jul 24, 2023

View reviewed changes

xarray/tests/test_backends.py Outdated Show resolved Hide resolved

RKuttruff and others added 3 commits July 24, 2023 14:31

Update xarray/tests/test_backends.py

3e75371

Co-authored-by: Deepak Cherian <[email protected]>

Alternate chunking method if dask not present in test run

7889f5a

[pre-commit.ci] auto fixes from pre-commit.com hooks

79773c0

for more information, see https://pre-commit.ci

Illviljan reviewed Jul 25, 2023

View reviewed changes

xarray/backends/zarr.py Outdated Show resolved Hide resolved

Improvements

11f3ffc

- Default value of None which will fall back to encoding specified behavior or Zarr defaults - If param (!= None) and encoding setting disagree, a ValueError is raised - Test case checks for compatible zarr version - Documented minimum required Zarr version

RKuttruff requested review from dcherian and Illviljan July 25, 2023 20:47

rileykk added 2 commits July 25, 2023 14:59

Bumped minimum zarr dependency version 2.10 -> 2.12

5bdbecf

Simplified test case

0083914

Zarr minimum dependency bump should make the version check no longer necessary

github-actions bot added CI Continuous Integration tools dependencies Pull requests that update a dependency file labels Jul 25, 2023

dcherian reviewed Jul 25, 2023

View reviewed changes

Illviljan reviewed Jul 25, 2023

View reviewed changes

rileykk and others added 3 commits July 25, 2023 15:33

Fixed table in docs

cba1276

Update xarray/backends/zarr.py

89c941b

Co-authored-by: Illviljan <[email protected]>

Update xarray/backends/zarr.py

860742c

Co-authored-by: Illviljan <[email protected]>

RKuttruff and others added 3 commits July 25, 2023 15:41

Update xarray/tests/test_backends.py

075f1fe

Co-authored-by: Illviljan <[email protected]>

Update xarray/tests/test_backends.py

8464b33

Co-authored-by: Illviljan <[email protected]>

Update xarray/backends/zarr.py

b6d611e

Co-authored-by: Illviljan <[email protected]>

RKuttruff requested a review from Illviljan July 25, 2023 23:38

Fix typing

4192241

Illviljan approved these changes Jul 26, 2023

View reviewed changes

jhamman reviewed Jul 27, 2023

View reviewed changes

Merge branch 'main' into write-empty-chunks

6dbf4a5

github-actions bot removed CI Continuous Integration tools dependencies Pull requests that update a dependency file labels Jul 27, 2023

Update xarray/core/dataset.py

b6919c0

Co-authored-by: Joe Hamman <[email protected]>

RKuttruff mentioned this pull request Jul 27, 2023

Allow setting of write_empty_chunks for zarr.core.Array objects zarr-developers/zarr-python#1478

Closed

rileykk and others added 3 commits July 27, 2023 12:14

Avoid setting of protected attribute in zarr Array object

ceae9bc

Merge remote-tracking branch 'origin/write-empty-chunks' into write-e…

f369cc5

…mpty-chunks

Merge branch 'main' into write-empty-chunks

ad0e0d4

RKuttruff requested review from d-v-b and jhamman July 27, 2023 20:04

Merge branch 'main' into write-empty-chunks

ac49a8f

jhamman approved these changes Aug 3, 2023

View reviewed changes

dcherian merged commit 0343761 into pydata:main Aug 4, 2023
27 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zarr : Allow setting `write_empty_chunks` #8016

Zarr : Allow setting `write_empty_chunks` #8016

RKuttruff commented Jul 24, 2023

welcome bot commented Jul 24, 2023

RKuttruff commented Jul 25, 2023

Illviljan commented Jul 25, 2023

Illviljan commented Jul 25, 2023

RKuttruff commented Jul 25, 2023

dcherian Jul 25, 2023

RKuttruff Jul 25, 2023

rabernat Jul 25, 2023

RKuttruff Jul 25, 2023

Illviljan Jul 25, 2023

RKuttruff Jul 25, 2023

jhamman Jul 27, 2023

d-v-b Jul 27, 2023

RKuttruff Jul 27, 2023

jhamman left a comment

jhamman Jul 27, 2023

RKuttruff commented Jul 27, 2023

RKuttruff commented Jul 27, 2023 •

edited

Loading

RKuttruff commented Jul 27, 2023

dcherian commented Aug 3, 2023

welcome bot commented Aug 4, 2023

dcherian commented Aug 4, 2023

RKuttruff commented Aug 4, 2023

Zarr : Allow setting write_empty_chunks #8016

Zarr : Allow setting write_empty_chunks #8016

Conversation

RKuttruff commented Jul 24, 2023

welcome bot commented Jul 24, 2023

RKuttruff commented Jul 25, 2023

Illviljan commented Jul 25, 2023

Illviljan commented Jul 25, 2023

RKuttruff commented Jul 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhamman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RKuttruff commented Jul 27, 2023

RKuttruff commented Jul 27, 2023 • edited Loading

RKuttruff commented Jul 27, 2023

dcherian commented Aug 3, 2023

welcome bot commented Aug 4, 2023

dcherian commented Aug 4, 2023

RKuttruff commented Aug 4, 2023

Zarr : Allow setting `write_empty_chunks` #8016

Zarr : Allow setting `write_empty_chunks` #8016

RKuttruff commented Jul 27, 2023 •

edited

Loading