DVC runs faster on system with more cores, how do I replicate this on a system with few cores?

[alita-moore]

I have a lot of files to save in dvc, and it seems that when I run commands such as dvc add on a system with 200 cores / threads it adds data at about 500-1000 files / second, but if I run the same command on a computer with 16 cores / threads it adds at a about 50 file/s. I tried to change the core.checksum_jobs config but it didn't change anything.

Basically, how can I get speed to be equivalent or better without having to use a 200 core machine (which is expensive) to do basic dvc commands?

NOTE: the cpu usage is low in both cases, so I'm guessing this has something to do with multi-threading?

[shcheklein]

@alita-moore are you running it on different cloud instances? can it be IO performance difference?

can you check the number of threads running btw? I think ps -T -p <pid> should do the trick.

[alita-moore]

It's the same instance just with different specs (only difference is number of cores, it's the same generation / architecture). Sure, I'll try that the next time I'm on the larger machine. Is there a way to configure the number of threads used by dvc? that would be super helpful. I noticed that when calculating the hashes the cpu is not being used very much. I wonder if it'd be possible to max out the cpu usage or something along those lines to not waste time.

[shcheklein]

Could you also check disk performance please to see if reading those files is a bottleneck. Also, yes, please check the effective number of threads that you are already getting with core.checksum_jobs.

[alita-moore]

I'm working on the machine again, so will update you on the outputs soon. I will say that dvc pull is extremely slow (about 50 files / second; which for reference is about 4 times slower that training my model). Here's the output for the relevant threads:

ps -T -p 44911
    PID    SPID TTY          TIME CMD
  44911   44911 pts/0    00:01:14 dvc
  44911   44919 pts/0    00:00:00 dvc
  44911   44952 pts/0    00:00:00 dvc
  44911   44956 pts/0    00:00:00 dvc
  44911   44987 pts/0    00:00:00 dvc

noting this is referencing the following command /usr/local/bin/python /usr/local/bin/dvc pull 1.0.0.dvc --no-run-cache --allow-missing --force

note that this is running on a machine with 128 vCPUs so the fact it has 5 threads running seems to be under-utilizing what's available. I'll be pushing the next few hours so I'll update you with the outputs of the push command.

[alita-moore]

by the way, I've noticed this thread. I think that it would make a lot of sense if dvc managed the compressiong and decompression of large amounts of files to reduce the actually number of files uploaded to s3. This would speed things up dramatically and also reduce my S3 bill. I think in the short term here I'll just compress / archhive myself, but it's worth noting that doing this adds a lot of overhead and is not ideal.

[skshetry]

Hi. We only hash one single file at a time, so checksum_jobs is unused. The performance difference that you are likely seeing is during object.transfer (which transfers to the cache) and checkout (checkout to workspace) where we use 4 * cpu_count number of jobs. So, that'd be 200 * 4 = 800 active file transfers. That is too many operations for dvc to handle, but unfortunately that is not configurable for dvc add.

[skshetry]

Hi, @alita-moore. You have provided very little information here about your setup/system and dataset sizes/shapes for me to help.

[skshetry]

If they are small files (<1MB), Python introduces high overhead that more threads are not going to help.

[alita-moore]

Sorry about that, to offer more insight, we are using an ec2 instance with 8 cores to save about 10 million images using dvc. Each image is roughly 0.5-4MB in size. Is there any more information I can give that would helpful?

[skshetry]

What is the config that you are using? What kind of filesystem and storage device (SSD) is it? Would it be possible to share dvc doctor output (from inside the repo)?

[alita-moore]

output from dvc doctor:

DVC version: 3.51.2 (pip)
-------------------------
Platform: Python 3.10.13 on Linux-6.8.0-1010-aws-x86_64-with-glibc2.36
Subprojects:
        dvc_data = 3.15.1
        dvc_objects = 5.1.0
        dvc_render = 1.0.2
        dvc_task = 0.4.0
        scmrepo = 3.3.6
Supports:
        http (aiohttp = 3.9.5, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.5, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2024.6.1, boto3 = 1.34.131)
Config:
        Global: /root/.config/dvc
        System: /etc/xdg/dvc
Cache types: symlink
Cache directory: ext4 on /dev/nvme1n1
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme1n1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/fe43d50b8d2357b489957043a91dd310

dvc config:

[core]
    autostage = true
    remote = datastore
    checksum_jobs = 16
['remote "datastore"']
    url = s3://bucket

the file system is an sc1 hard disk (or equivalent).

[skshetry]

@alita-moore, we have done some optimizations on dvc==3.54.0. It would be great if you could try.

Also, we have re-added support for core.checksum_jobs, so you can change the amount of cores used for hashing. However, note that, only large files are going to be hashed concurrently (>1MB). Small files are still going to be hashed sequentially.

[alita-moore]

beautiful, I'll check it out. Though I changed my approach to instead zip my files for faster processing. Thank you!