multithreaded concurrent downloads?

[Betonhaus]

Would it be possible to add the ability to download multiple files simultaneously when doing bulk downloads? So if one thread is stuck on a really big and/or slow file the other threads can knock through the other downloads while the one thread is chugging away?
Downloading all the files at once using xargs will bog down the system, but if there was a way to define a maximum number of threads so it only downloads 10 files at once instead of making 1,000 unique threads would be much faster.

[JustAnotherArchivist]

Duplicate of #412

Some implementations of xargs (e.g. GNU's) have a -P option to specify the concurrency. There's also GNU Parallel and others.

[Betonhaus]

is there a way to get ia to lock the file it's downloading? when downloading an archive if it's going too slow I'll open up additional tabs and download parts (eg: when downloading get-smart I started the initial download then later started ones that only downloaded one season at a time)

Eventually the first download will catch up to the other downloads, skip the files that they downloaded, then start downloading the exact same file that a different thread is downloading. Ideally it should see that specific file is being downloaded then skip to the next one. cancelling one thread causes the file to get deleted/corrupted

[vxbinaca]

try "parallel" if you're doing multiple items, but if it's a single item you're SOL