WISH: on_start() "hook" function to tweak settings before launching each check

[HenrikBengtsson]

Background

Some packages fail sporadically due to race conditions when "old" and "new" are checked in parallel. Common reasons are hard-coded TCP ports and shared file caches. We may even have race conditions between packages checked if they share the same dependency.

When it comes to reverse-dependency checking, I consider these type of errors false positives. Resolving them requires manual inspection and often rerunning the revdep checks for those packages sequentially. This is a tedious and potentially error-prone process.

Wish

Add a mechanism for calling a "hook" function just before R CMD check is launched. When revdep_check() calls this function, it could pass the name of the package checked and whether 'old' or 'new' is checked. This would allow me to do things like:

revdep_check(on_start = function(pkg, version = c("old", "new"), ...) {
  ## Use a fresh BiocFileCache folder specific to the current check
  path <- file.path(tempdir(), pkg, version)
  dir.create(path, recursive = TRUE)
  Sys.setenv(BFC_CACHE = path)
})

[gaborcsardi]

How about rerunning the failing checks with a single CPU? This is what I usually do if there are any failing checks:

revdepcheck::revdep_add_broken()
revdepcheck::revdep_check(num_workers = 1)

[HenrikBengtsson]

Yes, that's what I do, but it's tedious and time-consuming, especially if there are lots of packages. It is also forces you to recheck packages that are truly broken.

Related: Does revdep_add_broken() pick up packages with only +1 ERRORs, or does it also pick up those with -1? What about 1 ERRORs? For these types of race conditions, it's random whether "old" or "new" fails, and sometimes both fails.