async chunk writing

[kongdd]

Just wondering whether Zarr.jl supports parallel writing?
If yes, could you give one small example?

[meggart]

I am not really sure what the context is. It is completely fine if multiple processes write simultaneously into the same Zarr dataset, but it is up to the user/library to make sure that different processes/threads never write into the same chunk. So the following is fine:

using Zarr
a = zzeros(Float64,10,10,chunks=(10,1),path=tempname())
@sync for i=1:10
  @async a[:,i] = rand(10)
end
a[:,:]

However, when using multiple threads there may be problems coming from the different compression/decompression libraries, which are not thread-safe, so be careful when using compression.

Otherwise when writing across several chunks in a single call to setindex!, it will depend on the backend if chunks are written sequentially or simulaneously. There is a storage backend trait which is set to SequentialRead by default, but several backends like HTTP or S3 do concurrent reads and writes by default, so if you do

a[:,:] = rand(10,10)

it will depend on the backend if you do concurrent writes, but compression will always happen sequentially because of the problems with the underlying C libraries mentioned above.

[bjarthur]

However, when using multiple threads there may be problems coming from the different compression/decompression libraries, which are not thread-safe, so be careful when using compression.

perhaps it's time to add Blosc2 support to Zarr? i believe that is thread safe, but could be wrong.