Use dvc to track my data generation pipeline

[ZuoyunZheng]

Hi,
as the title suggest, i have a pipeline that depended on some data resources (tracked normally with dvc) to generate the data that i want to use for my seperate model training pipeline. I would like dvc to track my generated data. However it was not allowing me to with the error message:

ERROR: cannot update 'generated_data': overlaps with an output of stage: 'dvc.yaml:data_gen_stage' in 'dvc.yaml'
Run the pipeline or use 'dvc commit' to force update it.

I have since used dvc commit to explicitly track this "artifect" (not sure if this is the correct term).

I have seen the discussion here however i don't quite understand the rationale behind why dvc not allowing for explicit tracking of the output of a stage similar to data tracking? Secondly is there any implication to my forcibly doing dvc commit? Any help is appreciated. Thanks in advance! :)

[dberenbaum]

Hi @ZuoyunZheng! If the generated data is part of the outs: of your stage in dvc.yaml, it's already being tracked. You can see that by examining dvc.lock. It's just that dvc tracks these stage outputs slightly differently, tracking them whenever you do dvc repro or dvc exp run and saving the info in dvc.lock instead of .dvc files. You should not need to do dvc add or dvc commit.