Drop --live and corresponding dvc.yaml section?

[dberenbaum]

See iterative/dvclive#60 for context. Right now, dvclive integration requires a live section in dvc.yaml, and dvclive.init() is optional in the Python script for the stage. This might be problematic for a few reasons:

1. dvclive.init() directory may conflict with --live directory.
2. Without dvclive.init(), the script will only run through DVC, not if executed as python train.py directly.
3. --live introduces an option into DVC that is language-specific since dvclive is a Python library.

@pared has suggested that it might make sense to reverse the current implementation and instead:

• Require dvclive.init() even in DVC
• Drop --live and the corresponding dvc.yaml section
• Have DVC automatically integrate dvclive functionality where dvclive.init() is called in a stage

@pared Please correct any misinformation or add your thoughts.

[pared]

@dberenbaum

Drop --live and the corresponding dvc.yaml section

I wouldn't say I am for dropping live output, as it consists of plots dir and optionally metrics file, which by itself is problematic to handle, but currently works. Also, without live output, we would have to reconsider generation of training report - live helps to store information whether dvc should generate it or not.

Require dvclive.init() even in DVC

Yes, I think this should be required in order to not break users script just for sake of convenience. What might make sense is actually override init call with --live provided path. (credit for the idea to @jorgeorpinel). That way user is able to reproduce with dvc repro, dvc is happy with output, and if user wants to call python train, sure, why not.

Have DVC automatically integrate dvclive functionality where dvclive.init() is called in a stage

Yes, I think about that, though that would require some research - first problem I see is our verification of outs existence, which happens before we execute the script. If we were to implement it, we would be editing stage during execution.

So I guess the main point here is precedence of init over --live, I would vote for --live to be more important, and requiring to call init always.

[pared]

One more thought on dropping live - maybe we could do that.

One of the reasons of creating live output was to provide support for special HTML (avialable when dvc live show) is called that takes into consideration the optional metrics file and visualizes it too. If we, instead of treating live as separate entity, would treat it as plots directory, and add 2 flags to plots config (is_live and live_generate_report), we would get rid of live output and retain current behaviour. We would get rid of currently existing problem: that metrics file needs to be stored in git.

That would require one change on dvclive side. Instead of creating optional metrics file alongside plots directory, we would need to store it inside the directory (for example as .latest.json).

[dberenbaum]

treat it as plots directory

Do you mean that the .tsv files generated by dvclive would be tracked as plots by DVC?

add 2 flags to plots config (is_live and live_generate_report)

Could you explain what each flag would do? It seems that we need at least one flag to specify that the plots should be generated automatically at each step instead of requiring users to run dvc plots show to refresh.

We would get rid of currently existing problem: that metrics file needs to be stored in git.

Could you explain why?

That would require one change on dvclive side. Instead of creating optional metrics file alongside plots directory, we would need to store it inside the directory (for example as .latest.json).

Again, could you explain why?

[jorgeorpinel]

Oh shoot there's treads here 🤯

avialable when dvc live show) is called

I didn't realize there were plans for a dvc live command. Is there an issue we can refer too?

treat it as plots directory, and add 2 flags to plots config

Or could Dvclive have its own config files (which DVC can auto-detect)? To keep the integration less coupled

[dberenbaum]

I didn't realize there were plans for a dvc live command. Is there an issue we can refer too?

It's obviously not documented, but I realized you can use dvc live show and dvc live diff already. It seems that these are pretty much the same as dvc plots.

[jorgeorpinel]

Always requiring dvclive.init() seems fine and improves code consistency.

Have DVC automatically integrate dvclive functionality where dvclive.init() is called in a stage

Yes, I think about that, though that would require some research - first problem I see is our verification of outs existence, which happens before we execute the script. If we were to implement it, we would be editing stage during execution.

But this part seems pretty tricky indeed. But if there's a good non-hacky solution, then 👍 — could maybe also be used to detect checkpoints without requiring the checkpoint: true field in stage outputs (unrelated, sorry) so maybe @pmrowla has thought about how to implement something like this?

[pmrowla]

I don't really see any way straightforward way of automatically determining that an output should be a checkpoint. The user has to tell DVC to that these outputs should be handled in a special way - for checkpoints, they get unique behavior (like with exp run --reset) that should only apply to those specific outputs. It's the same as where we can't automatically identify a persist output, the user has to explicitly tell DVC that the output needs to be persistent.

---

WRT live, my current understanding is that currently for live DVC outputs it means that:

• <output_name> itself is a directory containing dvclive .tsv logs, and is treated like any other DVC-tracked persist output directory
• During stage execution, DVC will take some steps when signaled by dvclive:
  1. If the live output has summary = true, we will look for <output_name>.json, which is the summary file for this particular iteration, and track it with git (I think this is mainly meant to be used in conjunction with checkpoints?)
  2. If the live output has html = true, we will generate the summary plots <output_name>.html, which is not git or DVC tracked

My thoughts (based on that understanding) on how to get rid of the live outputs from dvc.yaml:

What about adjusting the signaling method so that the live signal file also contains the paths to the dvclive generated metrics? So if dvclive is init'ed with dvclive.init("path/to/training_metrics"), dvclive would generate signal files containing something like

summary: path/to/training_metrics.json
html: path/to/training_metrics.html

DVC would then just always monitor for the live signaling when running any stage, as opposed to only monitoring specific stages containing live outs. When signaled, if summary is present dvc would track the specified file with git. If html is present, dvc would generate the specified plots. So essentially this is just moving the live output section out of dvc.yaml and into the dvclive<->dvc signal file.

The main problem here would be that path/to/training_metrics itself is no longer automatically DVC-tracked. In theory, we could also modify the stage here when needed and automatically track path/to/training_metrics as a persist out (assuming the directory is not already tracked). We would also modify dvc.yaml at this time. However, I'm not sure that modifying pipeline stages during execution like this is something we would actually want to consider?

---

Now that I'm thinking about it, I suppose that in theory this might also work for checkpoints, but not as well as it would for dvclive.

Since we have full control over dvc and dvclive (and dvclive is python only), we can can make the signal files take any format we want, everything regarding signaling is completely hidden from the user.

Whereas for checkpoints we need to support signaling from any arbitrary (non-python) user stages, and requiring a simple "create file, wait for file to be removed" step is much more straightforward than, "generate file containing some specifically formatted list of files that DVC should treat as checkpoint outs". (And note that for checkpoints, we do need that specific list of files, since they need to be treated in a unique way upon exp run --reset.)

[jorgeorpinel]

automatically determining that an output should be a checkpoint
same as where we can't automatically identify a persist output

Yep @pmrowla sorry again, that is kind of unrelated to the OP, but I wasn't suggesting to avoid the checkpoint or persist fields, just to always listen for dvclive.init(), or make_checkpoint() for that matter (not sure), in any Python code run by DVC (regardless of dvc.yaml fields).

In the case of Dvcilve that could mean that we don't need the live field (as all the needed info is set by init()).

DVC would then just always monitor for the live signaling when running any stage, as opposed to only monitoring specific stages containing live outs

➕

[jorgeorpinel]

path/to/training_metrics itself is no longer automatically DVC-tracked
modifying pipeline stages during execution...

I didn't realize DVC auto-tracks the live logs dirs... If so then it may make sense to keep it in dvc.yaml. Can you confirm @pared? (We may need to update https://dvc.org/doc/dvclive/dvclive-with-dvc accordingly to clarify this.)

if dvclive is init'ed with dvclive.init("path/to/training_metrics"), dvclive would generate signal files...

May not be needed if init() is always required? Assuming every time the code runs you get the same init config.

essentially this is just moving the live output section out of dvc.yaml and into the dvclive<->dvc signal file
in theory this might also work for checkpoints, but not as well... generate file containing some specifically formatted list of files that DVC should treat as checkpoint outs

That's not my suggestion but sorry yet again @pmrowla, I shouldn't have complicated this discussion with that. Will think through and move to #5530. Please forget exp-related stuff here everyone!

[pmrowla]

Just to be clear, my opinion on

We would also modify dvc.yaml at this time. However, I'm not sure that modifying pipeline stages during execution like this is something we would actually want to consider?

is that we should avoid doing this (retroactively modifying pipeline a stage (dvc.yaml) during stage execution).

[dberenbaum]

I have a few follow-up questions:

1. How does DVC know to track the summary as a metrics file?
2. If there is a checkpoint output in the stage (cc @pmrowla):
   a. DVC generates a checkpoint at each dvclive.next_step(), even without a call to make_checkpoint(). How is this implemented?
   b. If dvclive.init() is included, it seems that checkpoints are no longer created at each dvclive.next_step(). Is this expected?

[pmrowla]

1. @pared can confirm but iirc, git-tracking the summary .json is just a hard coded step during the DVC callback that gets triggered by dvclive

2. dvclive calls make_checkpoint itself internally if the MetricLogger was inited with checkpoint=True. This is only done when we auto-init dvclive via DVC (for live outputs). When calling dvclive.init() manually, the MetricLogger checkpoint flag is not set.
   https://github.com/iterative/dvclive/blob/25f800bee3780f51539a260c04191daf17fcbd97/dvclive/metrics.py#L106

[dberenbaum]

1. @pared can confirm but iirc, git-tracking the summary `.json` is just a hard coded step during the DVC callback that gets triggered by dvclive

How does dvc in general know what to track as metrics when running commands like dvc metrics show? I guess I'm unclear what the requirements are for dvc to recognize a metrics file.

[pmrowla]

The file itself has to be tracked as a metrics output, but other than that we just treat the entire file contents as metrics as long as it is organized as a json/yaml dictionary with numeric values: https://dvc.org/doc/command-reference/metrics#supported-file-formats

[dberenbaum]

The file itself has to be tracked as a metrics output

Sorry, I keep missing something here. How is this handled internally? I'm still unclear how this works for the dvclive summary file, since it's not specified as a metrics output in dvc.yaml.

[pmrowla]

For regular metrics, you explicitly track the some_metric.json file as a DVC metrics output.

The live output is treated as a special case. Since we know that /path/to/output_name for live points to the dvclive logs directory, we also know that /path/to/output_name.json will point to the dvclive summary, and automatically handle it in dvc (like it is a metrics+cache: false output).

[dberenbaum]

1. `dvclive.init()` directory may conflict with `--live` directory.

Not only the directory, but also the values for summary and html in dvc.yaml might conflict.

Going back to the initial suggestion from @jorgeorpinel in iterative/dvclive#60 to drop the arguments to --live, Is it possible to simplify the dvc.yaml like:

stages:
  foo:
    live: true

I'm still unclear on whether dvc needs the directory location (and whether summary metrics are included) before calling the stage.

[dberenbaum]

How about hybrid approach? User could define live output as plots and we could use existing communication to get the information that it is actually live and not a plots, and add live, live_summary, live_html to plots config as needed?

Not sure I understand. Are you suggesting that the dvc.yaml look like:

stages:
  foo:
    cmd: python train.py
    metrics:
    - logs.json:
        cache: false
    plots:
    - logs/acc.tsv:
        cache: false

And train.py includes dvclive.init("logs", summary=True)?

Does your proposal address the concerns about conflicts between dvc.yaml and dvclive.init()? I'm struggling with how to resolve this conflict.

[dberenbaum]

DVC outputs are removed before running the stage, in order to avoid some unexpected behaviour/unnecessary assets in directory.

Can the outputs be removed when dvclive.init() is called?

[pared]

@dberenbaum

Does your proposal address the concerns about conflicts between dvc.yaml and dvclive.init()? I'm struggling with how to resolve this conflict.

Here are few things:

1. dvclive.init(path="logs") - logs should be defined in dvc.yaml too, if there is no match for plots in dvc.yaml - throw
2. summary - I believe summary should be moved inside the logs dir under .latest.json or something like that - we can program special behaviour "if live then treat logs/.latest.json as metric". When using plots we could consider live as special use case where we know to ignore .latest.json
3. html - well, i would define some sensible default ("dvclive_report.html") and allow user to change that.

Can the outputs be removed when dvclive.init() is called?

When running in DVC context - they cannot be removed by init call, because they do not exist - they should have been removed by DVC before the stage is executed.

init by itself removes the path if it exists.

[dberenbaum]

So are you proposing a dvc.yaml like this?

stages:
  foo:
    live: logs

1. `dvclive.init(path="logs")` - logs should be defined in `dvc.yaml` too, if there is no match for plots in `dvc.yaml` - throw

From a user perspective, it feels unnecessary and potentially conflicting to duplicate info in both dvc.yaml and dvclive. I guess it's okay, but it's not ideal.

When running in DVC context - they cannot be removed by init call, because they do not exist - they should have been removed by DVC before the stage is executed.

init by itself removes the path if it exists.

If dvc didn't know about the outputs because they aren't in dvc.yaml, can dvclive.init() remove them instead?

[pared]

From a user perspective, it feels unnecessary and potentially conflicting to duplicate info in both dvc.yaml and dvclive. I guess it's okay, but it's not ideal.

You are right. Let's step back - initial problem was that we were afraid of users who use --live, don't use dvclive.init - maybe instead of throwing on no dvclive.init, we should just use some default dir?

If dvc didn't know about the outputs because they aren't in dvc.yaml, can dvclive.init() remove them instead?

dvclive already can do that. If path exists, and we don't provide resume - path will be removed.