-
Notifications
You must be signed in to change notification settings - Fork 271
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slow writes of Zarr files using S3Map #820
Comments
At a glance, it seems the code is doing
Clearly, when the intent is to overwrite, these are unnecessary, and some seem to query the path just written. The only thing you might want to do is rm(recursive), if you know you are writing a whole dataset from scratch. s3fs can probably not do anything about this; whils you can prospectively pre-list a directory tree, because there are write ops in there too, the filecache will be regularly invalidated. Perhaps something clever could be done for updating an already-cached listing. I am not immediately sure if xarray or zarr is making the calls. It would be interesting to see what zarr alone does. I tried the following:
and probably we see the whole behaviour here. It's also worth noting that zarr is processing each variable/array sequentially, whereas these tiny metadata files can all be sent concurrently, as Maybe fsspec could provide a way to combine "caching" and "transactions" so that the effect of local-write->put() is conveniently achieved. |
Thanks for the detailed response @martindurant. The same requests are happening from Xarray even with an empty bucket to begin with.
Yes! Your minimal example captures the same log outputs as xr.to_zarr(), so I think we can focus on the interaction of Zarr and S3FS. I'm not sure why 'list', 'get', 'head' are necessary for the mode='w' case? Shouldn't there just be 'puts'? Would this require a change in s3fs or Zarr?
Interesting, thanks for clarifying! A streamlined |
I tried last night to write some code around simplecache to be general for any backend, but I don't think it's doing the right thing - partly because zarr still wants to list and read from the target, where the new files don't yet exist. It might be better to write better Transaction class(es) that know what to do with filesystems that have bulk put/commit operations (cc fsspec/filesystem_spec#1416 ), but they need to be read/write. It's also worth pointing out that zarr already supports separating the matadata mapper from the data mapper, so one could write a specialised deferred write thing that wraps FSStore. |
fsspec/filesystem_spec#1434 I think goes some way to resolving the problem: it caches writes during a transaction, putting them to remote in one concurrent operation, and rereading from local in the meantime. It does not address directory listing. |
I'm noticing an order of magnitude difference in writing a zarr file to an S3 bucket using
xarray.to_zarr(store=S3Map)
(~10s) versuss3.put(local_store, remote_store)
(~1s)The data is public, simple demonstration here
https://gist.github.com/scottyhq/effa642f00112971e2350d921a0aed9d
I notice the s3fs logs via Xarray have a lot of
head_object
andget_object
calls compared to onlyput_object
if usings3.put
with a local copy.I realize this may be considered an Xarray or Zarr Issue. However, I'm interested in interpreting this s3fs log and whether there are non-default settings to s3fs.S3FileSystem() or s3fs.S3Map() that are most appropriate for writing.
Truncated log:
versions
The text was updated successfully, but these errors were encountered: