-
-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Iterating over zip_disk.frame to beat 'Possible truncation of >= 4GB file' #345
Comments
I am not quite sure what is the issue based on your description. The code for |
Here is the code that I have used:
My .zip file is 174gb and the file inside is about 800gb. My assumption is that it does not retrieve all the data, and stops after decompressing the zip at 4GB, am I right here? Because the dataset I get IS not that large for a 800gb file, which has about 1.5bn occurrences, I would retrieve about 5million. Which definitely is not the case as bird data would give me 5m+ for this range, so it must be cutting off at 4GB right? I downloaded the simple dataset from here: https://www.gbif.org/occurrence/download?occurrence_status=present |
This is the code for the function. See if you can run it line by line and figure out what's wrong. You might want to do
for debugging
|
I think it is an inherent problem with R where >4gb of decompressing a file is truncated instead, therefore, I was thinking of how to overcome this. I had thought that the solution was iterating over Something like, n+1 iterations though I wouldn't be sure how to tell the programme that 'this portion of data has already been decompressed - skip this and do the next 4gb' |
I know that this function can bypass the unzip limit:
So passing this instead of |
nice. I will keep that in mind |
I keep getting the issue when unzipping a 174gb .zip file:
I know that it's an R problem, however, is there a way around this? otherwise, can something like looping over the .zip decompression until the until file is complete? Something similar to the in_chunks part of the function.
Where 4gb is decompressed, stored as a .tz and another 4gb decompressed and stored as a .tz etc ... while continuing from the previous decompression, until complete?
The text was updated successfully, but these errors were encountered: