Making DVC Repo API Public

[dberenbaum]

We are developing some specific Python APIs to meet needs that the CLI doesn't (#6850), but users often want to mimic the DVC CLI inside Python. In that case, we direct them to use dvc.repo.Repo, which we have often noted as not yet released.

This is a really common request and suggestion! See previous discussions and requests for background:

• How do I open a DVC-tracked directory from Python API ? #7379
• Expand Python API #5999
• Python API #5050
• Can't find documentation for calling dvc from python module. #2690
• list metrics files only #7133 (comment)
• https://discord.com/channels/485586884165107732/563406153334128681/916333985804734464
• https://discord.com/channels/485586884165107732/485596304961962003/909810821637472307
• https://discord.com/channels/485586884165107732/485596304961962003/907806666869842031
• https://discord.com/channels/485586884165107732/485596304961962003/902237057647271946
• https://discord.com/channels/485586884165107732/563406153334128681/895575067516338177
• https://discord.com/channels/485586884165107732/485596304961962003/872841035854778368
• https://discord.com/channels/485586884165107732/563406153334128681/870451499912790047
• https://discord.com/channels/485586884165107732/565699007037571084/859691975358808115
• https://discord.com/channels/485586884165107732/485596304961962003/856988029519396895
• https://discord.com/channels/485586884165107732/565699007037571084/842377740190613514
• https://discord.com/channels/485586884165107732/563406153334128681/834878080488177745
• https://discord.com/channels/485586884165107732/485596304961962003/816793023891243028
• https://discord.com/channels/485586884165107732/563406153334128681/810834803460014140
• https://discord.com/channels/485586884165107732/565699007037571084/803795760093200435
• https://discord.com/channels/485586884165107732/485596304961962003/798955806464147467
• https://discord.com/channels/485586884165107732/485596304961962003/783993894912262164
• https://discord.com/channels/485586884165107732/485596304961962003/783700824323784755
• https://discord.com/channels/485586884165107732/485596304961962003/764139396852482088
• https://discord.com/channels/485586884165107732/563406153334128681/758824048884580372
• https://discord.com/channels/485586884165107732/485596304961962003/731882559117721632
• https://discord.com/channels/485586884165107732/563406153334128681/729786984951971901
• https://discord.com/channels/485586884165107732/485596304961962003/704326952185626796
• https://discord.com/channels/485586884165107732/485596304961962003/698196937576284251
• https://discord.com/channels/485586884165107732/563406153334128681/660905790781259786
• https://discord.com/channels/485586884165107732/563406153334128681/605318413363183622
• https://discord.com/channels/485586884165107732/563406153334128681/596373813587869707
• https://discord.com/channels/485586884165107732/485586884165107734/549652969419964426

Way more users might find this helpful if they knew it existed, so what is preventing us from making this public?

• Documentation?
• Scope of dvc.repo.Repo?
• Abuse concerns?
• Maintenance concerns?
• Something else?

[iesahin]

I believe our command reference can also include API references if the API will have (almost) 1-1 correspondence with the CLI. There won't be duplicate efforts in that case. We'll simply list the API function along with the command, and the command options will be the arguments of these API functions.

[daavoo]

will have (almost) 1-1 correspondence with the CLI

For some commands, there is full 1-1 correspondence (almost all "data management/sync": add, checkout, commit, fetch, push, pull, remove, ...), but for others (an extreme case is exp show) there are flags that only apply to the CLI.

[daavoo]

A good first step in a similar direction would be to add docstrings to the Repo methods.

[jorgeorpinel]

I like this strategy at least as a first step. Add a block in or under Synopsis sections to point out the corresponding Repo method signature/usage. Esp. if it can be autogenerated from docstrings

[jorgeorpinel]

Idk what's prevented us from publishing it. We can find a solution for the documentation if the core team decides to make it public. Maintenance is also probably not a concern since we need to maintain it (code-wise) anyway, or if you meant maintaining the docs then again we can find a low-overhead solution e.g. simple automatic docs (rel. iterative/dvc.org#2770).

My Q is what would be the requirements to get to the point where we feel comfortable making it more official and visible. Complete docstrings? Refactoring something? I'm guessing it's mostly up to the core team for now. Thanks

[pmrowla]

Maintenance is also probably not a concern since we need to maintain it (code-wise) anyway

Maintenance is a concern though, because as soon as we officially say that we support using Repo as a stable public API, we need to account for maintaining backwards compatibility in non-major version DVC releases.

Right now, we are free to make changes as needed to something like Repo.pull() function arguments, or Repo.experiments.show() return values, since we only care about whether or not the rest of DVC internals in the current main version work with those changes. But once it is an officially supported API, we can't make those types of changes without breaking user code.

[daavoo]

Good points. I believe we should disccuss how and what should be exposed and what changes would be needed in code to avoid issues like these. I don't think we should directly expose all the internal Repo API as it is but I do think we should try to push towards consolidated Python API.

[daavoo]

I have remembered that I heavily used Repo API in https://github.com/iterative/dvc-streamlit but never made it public. We could make it public and use it as an excuse to showcase how the API can be used to build something on top.

[skshetry]

I'd prefer making the purpose of Repo API clearer first before making it public.

[daavoo]

• Data Collection

I believe one of the main purposes (not the only one) would be collecting data in a useful way.

repo.{experiments|metrics|params|plots}.show all return plain python types, with undocumented structure, and the name of show is somehow misleading.

From a Python API, I would expect something more in the lines of Repo.get_{experiments|metrics|params|plots} returning lists of specific types (we currently have common type Output for metrics, plots and outs) and sharing all the same few arguments, like:

def get_metrics(self, targets=None, revs=None) -> List[Metric]:

Because it's Python API I think it's not that bad if we don't provide more elaborated filtering because users can easily implement custom filtering in 1-2 lines

[daavoo]

This is my personal opinion but instead of having all over the Python API args like all_branches, all_tags, all_commits I would prefer to expose a flexible way of collecting those revisions separately and passing them as revs argument.

[skshetry]

repo.{experiments|metrics|params|plots}.show

Right, but that can be provided through dvc.api too right? Why make Repo API public which is quite broad?

AFAICT it was created so that we can wrap this in CLI and test it easily through Pythonic API instead of invoking CLI.
But as CLI is becoming more and more interactive and Repo API becoming more machine-readable, I just don't see enough value to keep doing this.

[skshetry]

So my questions would be:

1. What is the purpose of Repo API?
2. Should it mirror CLI and to what extent? Should it print any outputs to stderr/stdout?
3. Should it return a dict/retval/pythonic datastructure?
4. Should it be more Pythonic API mimicking CLI rather than being used internally for CLI?

At least, I don't think it'd be easy to use Repo API in CLI and also provide pythonic Repo API at the same time. So this is definitely going to increase maintenance burden.

[daavoo]

What is the purpose of Repo API?

Providing a single class that once instantiated serves as a single entry point for:

• collection of dvc data (deps, experiments, metrics, params, plots, outs) across multiple revisions
• data management operations (add, commit, checkout, move, remove, etc.)
• data sync operations that require repo status (fetch, pull, push)
• pipelines operations (dvc repro, dvc exp run), not sure about altering pipeline (i.e. stage add)
• status (diff, status)

Some things like the existing API methods or a traversable filesystem could live separately

Should it mirror CLI and to what extent?

I think it should support pretty much all the same operations but don't necessarily mirror the CLI.

For example in a repo.add Python API I wouldn't add --glob or --recursive options and would keep the operation granular. Users can do the globing and more flexible things in 1 python line interacting with our granular repo.add.

Should it print any outputs to stderr/stdout?

No strong opinion.

Should it return a dict/retval/pythonic datastructure?

I think pythonic class for collection operations, other cases probably better discussed individually.

Should it be more Pythonic API mimicking CLI rather than being used internally for CLI?

I think that internal Repo API and public Repo API could co-exists.

I think the discussion tends to focus about making the internal Repo API public because Repo feels like the right abstraction. I know the internal architecture of the Repo API has its drawbacks and people don't want to be tied to the current status by making it public.

I think we could create separate public Repo API that doesn't prevent us from refactoring/switching the internal abstraction

[daavoo]

Relevant discussion:

#7613 (comment)

[dberenbaum]

From discussion of that PR, I think we need some API guidelines (similar to https://clig.dev/).

Some suggested guidelines based on discussion so far:

• Purposes of API methods may be either to mimic the CLI if there is a use case where users might want to do something similar via Python (like api: Add params_show. #7613 if dvc params show actually existed), or to provide functionality that is useful in the API but doesn't make sense in the CLI (like dvc.api.open()).
• If providing similar functionality to the CLI, provide a "porcelain" method that uses and resembles the CLI implementation, but adjust output/UI to fit the user needs/expectations in Python (like flattening the output in api: Add params_show. #7613).

[dberenbaum]

Also, docstrings are useful for public API methods, and auto-generated docs would make it easier to develop and maintain.