How to manage external data efficiently?

[piseabhijeet]

Hi,
I have configured dvc for data version control in my Git project. I have some data on a Temp folder D:\Temp\Data.

I add it as an external dependency using the command:
dvc add --external D:\Temp\Data
which produces a Data.dvc file

I now define a dvc run which copies the data from D:\Temp\Data and keeps it in my Workspace. It produces a .yaml and .lock file.

I now change the original data on D:\Temp\Data by adding new files to it.

When i do a dvc status, it shows files have been changed, but i dont know how to add those the version control. I tried dvc import but it works on only DVC/Git based repos not on data on local storage or S3.

I do not want to push the data because it bloats the storage but i want to track the same using external dependency only.

Any help would be appreciated :)

[pmrowla]

To add the new version to DVC and update Data.dvc you would just rerun

dvc add --external D:\Temp\Data

(This is the same way that you update regular local workspace data which has been tracked via dvc add).

Using external data is a fairly advanced DVC feature and generally not recommended, can you share a bit more about your specific needs/use case here?

[piseabhijeet]

Oh okay, got it, thanks @pmrowla. I thought we need to create an output stage after an external data is specified. Then in that case there is no need to redefine pipelines again. Is there a good reference on how to use an external source efficiently?

[pmrowla]

There is really only the official documentation. Our support for external data is likely going to change, because a vast majority of the time what people are really looking for is #4520, which has already been mentioned. And in fact, I'm still fairly sure that feature is actually what you want here (it does what your original pipeline was trying to do).

[piseabhijeet]

@pmrowla
Thank you for the prompt response. Is there a timeline as to when this feature will be available?

Other thing, not sure if it is relevant, is there a support for dvc with mlflow? By support i mean currently dvc has data processing as well as experiment reproduction stages and mlflow also has the same, are there any best practices of using dvc as data version tool + mlflow as MLOps tool?

[pmrowla]

It's fairly high on our priorities list given that it is a frequently requested feature, but there is currently no estimate on when it will be available.

Regarding MLFlow, there are a few projects that use both DVC and MLFlow, one example is https://medium.com/ixorthink/our-machine-learning-workflow-dvc-mlflow-and-training-in-docker-containers-5b9c80cdf804, you'll have to look around for some others.

[piseabhijeet]

Than you @pmrowla , the discussion helped me to understand things in a better manner :)

[efiop]

@piseabhijeet Looks like you are looking for #4520 , which is on our todo list right now. The current workflow to achieve that is to just copy the file you wish to add into your workspace and then dvc add data there.

[piseabhijeet]

@efiop Thank you for the update, My scenario is explained in
#5154 (reply in thread)

I need to track data on S3 without duplicating it on S3 using dvc push (remote storage scenario) or on cache (external dependency scenario) and in the same .dvc workspace file