Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Proof of concept implementation for local package caching #268

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

tzakharko
Copy link

This is a proof of concept implementation for #261

The basic idea is to leverage the fact that the metadata of installed packages is already cached. So we don't need to do much — just inject some metadata describing the contents of the local repository (in this initial draft it's the local path + the file checksums) and do nothing if they match the already installed package. Seems to work well enough in local tests, but I cannot exclude that I missed something crucial — the package logic is fairly complex and I barely scraped the surface of the iceberg in trying to understand it.

The checksumming currently relies on tools::md5sum() and simply concatenates the hash for all files (just to show that it works), but we probably want to use digest here for proper sha256 (this would mean hard dependency on digest however). Otherwise we need to find a way to mix the md5 hashes somehow without introducing new package dependencies. Regarding performance: checksumming does introduce some overhead, but making a package is still much much slower, so if you local packages do not change too often, amortized savings over time can be substantial.

Please tell me what you think and I'll do my best to transition it to something more production ready.

Approach: relies on built-in caching for installed packages to store file checksums for local packages and use them to determine whether the contents have changed.
@tzakharko
Copy link
Author

Not sure why the pipeline is failing, there are some references to invalid GitHub credentials etc... all tests pass on my local machine (macOS 12.1 with R 4.1)

@tzakharko
Copy link
Author

@gaborcsardi Did you by any chance have the time to look at the patch and see whether this approach is in principle sufficient?

@gaborcsardi
Copy link
Member

I am sorry for the long wait. I am working on pkgdepends again, and will look at this soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants