Limit parallelism

[lgarron]

Thanks to #438 , I'm able to sync without slowing my network to a crawl! 🤩
It's helped me centralize all my backups on a new computer, which I'm grateful for.

Unfortunately, the only files left to sync are all several GB each. When I initially start Maestral, it tries to download a single one of those files, but soon enough Maestral tries to download them in parallel:

This means that it takes most of a day for any of the downloads to complete. But that rarely happens, because one of the files runs into a maestral.exceptions.DataCorruptionError error (I'm not sure why, but I wouldn't rule out a network issue) and then all the others do. At this point, it's getting more common for Maestral to crash completely, before any individual file is downloaded. So when I restart Maestral, it's always redoing the same work.

2023-02-10 05:01:53 sync INFO: Syncing ↓ 254/276
2023-02-10 06:32:19 sync INFO: Syncing ↓ 255/276
2023-02-10 08:30:59 sync INFO: Could not sync Waltz - 70D Source.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 08:30:59 sync INFO: Could not sync C0002.MP4
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 08:45:29 sync INFO: Could not sync 00111 - Feliks 1 of 2.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 08:56:56 manager INFO: Connecting...
2023-02-10 08:57:06 manager INFO: Connected
2023-02-10 09:00:09 sync INFO: Could not sync C0115.MP4
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:00:13 sync INFO: Could not sync C0113.MP4
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:00:25 sync INFO: Could not sync C0003.MP4
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:01:11 sync INFO: Could not sync Feliks Zemdegs - Full HD.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:01:16 sync INFO: Could not sync 00020 - Andy Smith.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:01:37 sync INFO: Could not sync 00032 - Sebastien Weyer.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:01:37 sync INFO: Could not sync 00006 - Kevin Costello.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:01:39 sync INFO: Could not sync 00007 - Cornelius and Kevin Costello.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:01:40 sync INFO: Could not sync 00004 - Rowe and Andrew Ricci.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:01:41 sync INFO: Could not sync 00021 - Andy Smith and Justin Mallari.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:01:42 sync INFO: Could not sync 00035 - Mats Valk 1 of 2.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:01:51 sync INFO: Could not sync 00110 - Phillip Weyer.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:02:07 sync INFO: Could not sync 00034 - Drew Brads.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:02:11 sync INFO: Could not sync 00003 - Rowe.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:02:11 sync INFO: Could not sync 00008 - Breandan.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:02:11 sync INFO: Could not sync 00009 - Kevin Hays.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:02:19 sync INFO: Could not sync 00033 - Eric Limeback.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:02:19 sync INFO: Could not sync 00023 - Seungbeom.mov
Traceback (most recent call last):
  File "app_packages/maestral/sync.py", line 3350, in _create_local_entry
  File "app_packages/maestral/sync.py", line 3447, in _on_remote_file
  File "app_packages/maestral/sync.py", line 3440, in _on_remote_file
  File "app_packages/maestral/client.py", line 360, in wrapper
  File "app_packages/maestral/client.py", line 336, in wrapper
  File "app_packages/maestral/client.py", line 788, in download
maestral.exceptions.DataCorruptionError: Data corrupted. Please retry download.
2023-02-10 09:02:19 manager INFO: Connection lost
2023-02-10 09:02:19 manager INFO: Shutting down threads...
2023-02-10 09:02:19 sync INFO: Sync aborted
2023-02-10 09:02:19 manager INFO: Paused
2023-02-10 09:02:19 manager INFO: Connecting...
2023-02-10 09:02:23 manager INFO: Retrying failed syncs...
2023-02-10 09:02:23 sync INFO: Syncing ↓ /selective sync/2021-06-14 | archived/opening videos/opening ceremony dslr/waltz - 70d source.mov
2023-02-10 09:08:41 manager INFO: Connecting...
2023-02-10 09:08:53 manager INFO: Connected
2023-02-10 09:09:05 manager INFO: Connecting...
2023-02-10 09:09:15 manager INFO: Connected
2023-02-10 09:09:48 manager INFO: Connecting...
2023-02-10 09:10:22 manager INFO: Connected
2023-02-10 09:10:44 manager INFO: Connecting...
2023-02-10 09:10:55 manager INFO: Connected
2023-02-10 12:05:37 daemon INFO: Starting daemon
2023-02-10 12:05:37 manager INFO: Connected
2023-02-10 12:05:38 manager INFO: Retrying failed syncs...
2023-02-10 12:05:38 sync INFO: Syncing ↓ /selective sync/2021-06-14 | archived/opening videos/opening ceremony dslr/waltz - 70d source.mov

For these files, I don't need any parallelism, and in fact there is no downside to downloading one at a time. Could I ask for a config for Maestral to limit the number of parallel downloads/uploads to see if it prevents these crashes?