Merge pull request #1222 from nf-core/dev

Dev -> Master for v2.1 release

.github/workflows/create-lint-wf.yml

@@ -53,7 +53,7 @@ jobs:
         run: nf-core --log-file log.txt lint --dir nf-core-testpipeline --fail-ignored --release
 
       - name: nf-core modules install
-        run: nf-core --log-file log.txt modules install fastqc --dir nf-core-testpipeline/ --force --latest
+        run: nf-core --log-file log.txt modules install fastqc --dir nf-core-testpipeline/ --force
 
       - name: Upload log file artifact
         if: ${{ always() }}

.github/workflows/sync.yml

@@ -67,7 +67,7 @@ jobs:
             --from-branch dev \
             --pull-request \
             --username nf-core-bot \
-            --repository nf-core/${{ matrix.pipeline }}
+            --github-repository nf-core/${{ matrix.pipeline }}
 
       - name: Upload sync log file artifact
         if: ${{ always() }}

CHANGELOG.md

@@ -1,5 +1,32 @@
 # nf-core/tools: Changelog
 
+## [v2.1 - Zinc Zebra](https://github.com/nf-core/tools/releases/tag/2.1) - [2021-07-27]
+
+### Template
+
+* Correct regex pattern for file names in `nextflow_schema.json`
+* Remove `.` from nf-core/tools command examples
+* Update Nextflow installation link in pipeline template ([#1201](https://github.com/nf-core/tools/issues/1201))
+* Command `hostname` is not portable [[#1212](https://github.com/nf-core/tools/pull/1212)]
+* Changed how singularity and docker links are written in template to avoid duplicate links
+
+### General
+
+* Changed names of some flags with `-r` as short options to make the flags more consistent between commands.
+
+### Modules
+
+* Added consistency checks between installed modules and `modules.json` ([#1199](https://github.com/nf-core/tools/issues/1199))
+* Added support excluding or specifying version of modules in `.nf-core.yml` when updating with `nf-core modules install --all` ([#1204](https://github.com/nf-core/tools/issues/1204))
+* Created `nf-core modules update` and removed updating options from `nf-core modules install`
+* Added missing function call to `nf-core lint` ([#1198](https://github.com/nf-core/tools/issues/1198))
+* Fix `nf-core lint` not filtering modules test when run with `--key` ([#1203](https://github.com/nf-core/tools/issues/1203))
+* Fixed `nf-core modules install` not working when installing from branch with `-b` ([#1218](https://github.com/nf-core/tools/issues/1218))
+* Added prompt to choose between updating all modules or named module in  `nf-core modules update`
+* Check if modules is installed before trying to update in `nf-core modules update`
+* Verify that a commit SHA provided with `--sha` exists for `install/update` commands
+* Add new-line to `main.nf` after `bump-versions` command to make ECLint happy
+
 ## [v2.0.1 - Palladium Platypus Junior](https://github.com/nf-core/tools/releases/tag/2.0.1) - [2021-07-13]
 
 ### Template

README.md

@@ -28,7 +28,8 @@ A python package with helper tools for the nf-core community.
     * [`modules list` - List available modules](#list-modules)
         * [`modules list remote` - List remote modules](#list-remote-modules)
         * [`modules list local` - List installed modules](#list-installed-modules)
-    * [`modules install` - Install or update modules in pipeline](#install-or-update-modules-in-a-pipeline)
+    * [`modules install` - Install modules in a pipeline](#install-modules-in-a-pipeline)
+    * [`modules update` - Update modules in a pipeline](#update-modules-in-a-pipeline)
     * [`modules remove` - Remove a module from a pipeline](#remove-a-module-from-a-pipeline)
     * [`modules create` - Create a module from the template](#create-a-new-module)
     * [`modules create-test-yml` - Create the `test.yml` file for a module](#create-a-module-test-config-file)
@@ -887,7 +888,7 @@ By default, the tool will collect workflow variables from the current branch in
 You can supply the `--from-branch` flag to specific a different branch.
 
 Finally, if you give the `--pull-request` flag, the command will push any changes to the remote and attempt to create a pull request using the GitHub API.
-The GitHub username and repository name will be fetched from the remote url (see `git remote -v | grep origin`), or can be supplied with `--username` and `--repository`.
+The GitHub username and repository name will be fetched from the remote url (see `git remote -v | grep origin`), or can be supplied with `--username` and `--github-repository`.
 
 To create the pull request, a personal access token is required for API authentication.
 These can be created at [https://github.com/settings/tokens](https://github.com/settings/tokens).
@@ -901,6 +902,13 @@ This allows multiple pipelines to use the same code for share tools and gives a
 
 The nf-core DSL2 modules repository is at <https://github.com/nf-core/modules>
 
+The modules supercommand comes with two flags for specifying a custom remote:
+
+* `--github-repository <github repo>`: Specify the repository from which the modules should be fetched. Defaults to `nf-core/modules`.
+* `--branch <branch name>`: Specify the branch from which the modules shoudl be fetched. Defaults to `master`.
+
+Note that a custom remote must follow a similar directory structure to that of `nf-core/moduleś` for the `nf-core modules` commands to work properly.
+
 ### List modules
 
 The `nf-core modules list` command provides the subcommands `remote` and `local` for listing modules installed in a remote repository and in the local pipeline respectively. Both subcommands come with the `--key <keywords>` option for filtering the modules by keywords.
@@ -962,7 +970,7 @@ INFO     Modules installed in '.':
 └─────────────┴─────────────────┴─────────────┴────────────────────────────────────────────────────────┴────────────┘
 ```
 
-### Install or update modules in a pipeline
+### Install modules in a pipeline
 
 You can install modules from [nf-core/modules](https://github.com/nf-core/modules) in your pipeline using `nf-core modules install`.
 A module installed this way will be installed to the `./modules/nf-core/modules` directory.
@@ -978,20 +986,77 @@ $ nf-core modules install
     nf-core/tools version 2.0
 
 ? Tool name: cat/fastq
-? Select 'cat/fastq' version: Rename software/ directory to modules/ to re-organise module structure ...truncated...
 INFO     Installing cat/fastq
 INFO     Downloaded 3 files to ./modules/nf-core/modules/cat/fastq
 ```
 
-You can pass the module name as an optional argument to `nf-core modules install` instead of using the cli prompt, eg: `nf-core modules install fastqc`.
+You can pass the module name as an optional argument to `nf-core modules install` instead of using the cli prompt, eg: `nf-core modules install fastqc`. You can specify a pipeline directory other than the current working directory by using the `--dir <pipeline dir>`.
 
-There are five flags that you can use with this command:
+There are three additional flags that you can use when installing a module:
 
-* `--dir <pipeline_dir>`: Specify a pipeline directory other than the current working directory.
-* `--latest`: Install the latest version of the module instead of specifying the version using the cli prompt.
 * `--force`: Overwrite a previously installed version of the module.
+* `--prompt`: Select the module version using a cli prompt.
+* `--sha <commit_sha>`: Install the module at a specific commit from the `nf-core/modules` repository.
+
+### Update modules in a pipeline
+
+You can update modules installed from a remote repository in your pipeline using `nf-core modules update`.
+
+```console
+$ nf-core modules update
+                                          ,--./,-.
+          ___     __   __   __   ___     /,-._.--~\
+    |\ | |__  __ /  ` /  \ |__) |__         }  {
+    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
+                                          `._,._,'
+
+    nf-core/tools version 2.0
+
+? Tool name: fastqc
+INFO     Updating 'nf-core/modules/fastqc'
+INFO     Downloaded 3 files to ./modules/nf-core/modules/fastqc
+```
+
+You can pass the module name as an optional argument to `nf-core modules update` instead of using the cli prompt, eg: `nf-core modules update fastqc`. You can specify a pipeline directory other than the current working directory by using the `--dir <pipeline dir>`.
+
+There are four additional flags that you can use with this command:
+
+* `--force`: Reinstall module even if it appears to be up to date
+* `--prompt`: Select the module version using a cli prompt.
 * `--sha <commit_sha>`: Install the module at a specific commit from the `nf-core/modules` repository.
-* `--all`: Use this flag to change versions on all installed modules. Has the same effect as running `nf-core modules install --force --latest` on all installed modules. To change all modules to a specific version you can run `nf-core modules install --all --sha <commit sha>`.
+* `--all`: Use this flag to run the command on all modules in the pipeline.
+
+If you don't want to update certain modules or want to update them to specific versions, you can make use of the `.nf-core.yml` configuration file. For example, you can prevent the `star/align` module installed from `nf-core/modules` from being updated by adding the following to the `.nf-core.yml` file:
+
+```yaml
+update:
+  nf-core/modules:
+    star/align: False
+```
+
+If you want this module to be updated only to a specific version (or downgraded), you could instead specifiy the version:
+
+```yaml
+update:
+  nf-core/modules:
+    star/align: "e937c7950af70930d1f34bb961403d9d2aa81c7"
+```
+
+This also works at the repository level. For example, if you want to exclude all modules installed from `nf-core/modules` from being updated you could add:
+
+```yaml
+update:
+  nf-core/modules: False
+```
+
+or if you want all modules in `nf-core/modules` at a specific version:
+
+```yaml
+update:
+  nf-core/modules: "e937c7950af70930d1f34bb961403d9d2aa81c7"
+```
+
+Note that the module versions specified in the `.nf-core.yml` file has higher precedence than versions specified with the command line flags, thus aiding you in writing reproducible pipelines.
 
 ### Remove a module from a pipeline
 

nf_core/__main__.py

@@ -209,7 +209,7 @@ def launch(pipeline, id, revision, command_only, params_in, params_out, save_all
 
 @nf_core_cli.command(help_priority=3)
 @click.argument("pipeline", required=False, metavar="<pipeline name>")
-@click.option("-r", "--release", type=str, help="Pipeline release")
+@click.option("-r", "--revision", type=str, help="Pipeline release")
 @click.option("-o", "--outdir", type=str, help="Output directory")
 @click.option(
     "-x", "--compress", type=click.Choice(["tar.gz", "tar.bz2", "zip", "none"]), help="Archive compression type"
@@ -223,15 +223,15 @@ def launch(pipeline, id, revision, command_only, params_in, params_out, save_all
     help="Don't / do copy images to the output directory and set 'singularity.cacheDir' in workflow",
 )
 @click.option("-p", "--parallel-downloads", type=int, default=4, help="Number of parallel image downloads")
-def download(pipeline, release, outdir, compress, force, container, singularity_cache_only, parallel_downloads):
+def download(pipeline, revision, outdir, compress, force, container, singularity_cache_only, parallel_downloads):
     """
     Download a pipeline, nf-core/configs and pipeline singularity images.
 
     Collects all files in a single archive and configures the downloaded
     workflow to use relative paths to the configs and singularity images.
     """
     dl = nf_core.download.DownloadWorkflow(
-        pipeline, release, outdir, compress, force, container, singularity_cache_only, parallel_downloads
+        pipeline, revision, outdir, compress, force, container, singularity_cache_only, parallel_downloads
     )
     dl.download_workflow()
 
@@ -349,15 +349,15 @@ def lint(dir, release, fix, key, show_passed, fail_ignored, markdown, json):
 ## nf-core module subcommands
 @nf_core_cli.group(cls=CustomHelpOrder, help_priority=7)
 @click.option(
-    "-r",
-    "--repository",
+    "-g",
+    "--github-repository",
     type=str,
     default="nf-core/modules",
     help="GitHub repository hosting modules.",
 )
 @click.option("-b", "--branch", type=str, default="master", help="Branch of GitHub repository hosting modules.")
 @click.pass_context
-def modules(ctx, repository, branch):
+def modules(ctx, github_repository, branch):
     """
     Tools to manage Nextflow DSL2 modules as hosted on nf-core/modules.
     """
@@ -367,7 +367,7 @@ def modules(ctx, repository, branch):
 
     # Make repository object to pass to subcommands
     try:
-        ctx.obj["modules_repo_obj"] = nf_core.modules.ModulesRepo(repository, branch)
+        ctx.obj["modules_repo_obj"] = nf_core.modules.ModulesRepo(github_repository, branch)
     except LookupError as e:
         log.critical(e)
         sys.exit(1)
@@ -421,18 +421,17 @@ def local(ctx, keywords, json, dir):
 @click.pass_context
 @click.argument("tool", type=str, required=False, metavar="<tool> or <tool/subtool>")
 @click.option("-d", "--dir", type=click.Path(exists=True), default=".", help="Pipeline directory. Defaults to CWD")
-@click.option("-l", "--latest", is_flag=True, default=False, help="Install the latest version of the module")
-@click.option("-f", "--force", is_flag=True, default=False, help="Force installation of module if it already exists")
+@click.option("-p", "--prompt", is_flag=True, default=False, help="Prompt for the version of the module")
+@click.option("-f", "--force", is_flag=True, default=False, help="Force reinstallation of module if it already exists")
 @click.option("-s", "--sha", type=str, metavar="<commit sha>", help="Install module at commit SHA")
-@click.option("-a", "--all", is_flag=True, default=False, help="Update all modules installed in pipeline")
-def install(ctx, tool, dir, latest, force, sha, all):
+def install(ctx, tool, dir, prompt, force, sha):
     """
-    Install/update DSL2 modules within a pipeline.
+    Install DSL2 modules within a pipeline.
 
     Fetches and installs module files from a remote repo e.g. nf-core/modules.
     """
     try:
-        module_install = nf_core.modules.ModuleInstall(dir, force=force, latest=latest, sha=sha, update_all=all)
+        module_install = nf_core.modules.ModuleInstall(dir, force=force, prompt=prompt, sha=sha)
         module_install.modules_repo = ctx.obj["modules_repo_obj"]
         exit_status = module_install.install(tool)
         if not exit_status and all:
@@ -446,6 +445,31 @@ def install(ctx, tool, dir, latest, force, sha, all):
 @click.pass_context
 @click.argument("tool", type=str, required=False, metavar="<tool> or <tool/subtool>")
 @click.option("-d", "--dir", type=click.Path(exists=True), default=".", help="Pipeline directory. Defaults to CWD")
+@click.option("-f", "--force", is_flag=True, default=False, help="Force update of module")
+@click.option("-p", "--prompt", is_flag=True, default=False, help="Prompt for the version of the module")
+@click.option("-s", "--sha", type=str, metavar="<commit sha>", help="Install module at commit SHA")
+@click.option("-a", "--all", is_flag=True, default=False, help="Update all modules installed in pipeline")
+def update(ctx, tool, dir, force, prompt, sha, all):
+    """
+    Update DSL2 modules within a pipeline.
+
+    Fetches and updates module files from a remote repo e.g. nf-core/modules.
+    """
+    try:
+        module_install = nf_core.modules.ModuleUpdate(dir, force=force, prompt=prompt, sha=sha, update_all=all)
+        module_install.modules_repo = ctx.obj["modules_repo_obj"]
+        exit_status = module_install.update(tool)
+        if not exit_status and all:
+            sys.exit(1)
+    except UserWarning as e:
+        log.error(e)
+        sys.exit(1)
+
+
+@modules.command(help_priority=4)
+@click.pass_context
+@click.argument("tool", type=str, required=False, metavar="<tool> or <tool/subtool>")
+@click.option("-d", "--dir", type=click.Path(exists=True), default=".", help="Pipeline directory. Defaults to CWD")
 def remove(ctx, dir, tool):
     """
     Remove a module from a pipeline.
@@ -459,11 +483,11 @@ def remove(ctx, dir, tool):
         sys.exit(1)
 
 
-@modules.command("create", help_priority=4)
+@modules.command("create", help_priority=5)
 @click.pass_context
 @click.argument("tool", type=str, required=False, metavar="<tool> or <tool/subtool>")
 @click.option("-d", "--dir", type=click.Path(exists=True), default=".", metavar="<directory>")
-@click.option("-a", "--author", type=str, metavar="<author>", help="Module author's GitHub username")
+@click.option("-a", "--author", type=str, metavar="<author>", help="Module author's GitHub username prefixed with '@'")
 @click.option("-l", "--label", type=str, metavar="<process label>", help="Standard resource label for process")
 @click.option("-m", "--meta", is_flag=True, default=False, help="Use Groovy meta map for sample information")
 @click.option("-n", "--no-meta", is_flag=True, default=False, help="Don't use meta map for sample information")
@@ -497,10 +521,10 @@ def create_module(ctx, tool, dir, author, label, meta, no_meta, force, conda_nam
         sys.exit(1)
 
 
-@modules.command("create-test-yml", help_priority=5)
+@modules.command("create-test-yml", help_priority=6)
 @click.pass_context
 @click.argument("tool", type=str, required=False, metavar="<tool> or <tool/subtool>")
-@click.option("-r", "--run-tests", is_flag=True, default=False, help="Run the test workflows")
+@click.option("-t", "--run-tests", is_flag=True, default=False, help="Run the test workflows")
 @click.option("-o", "--output", type=str, help="Path for output YAML file")
 @click.option("-f", "--force", is_flag=True, default=False, help="Overwrite output YAML file if it already exists")
 @click.option("-p", "--no-prompts", is_flag=True, default=False, help="Use defaults without prompting")
@@ -519,7 +543,7 @@ def create_test_yml(ctx, tool, run_tests, output, force, no_prompts):
         sys.exit(1)
 
 
-@modules.command(help_priority=6)
+@modules.command(help_priority=7)
 @click.pass_context
 @click.argument("tool", type=str, required=False, metavar="<tool> or <tool/subtool>")
 @click.option("-d", "--dir", type=click.Path(exists=True), default=".", metavar="<pipeline/modules directory>")
@@ -551,7 +575,7 @@ def lint(ctx, tool, dir, key, all, local, passed):
         sys.exit(1)
 
 
-@modules.command(help_priority=7)
+@modules.command(help_priority=8)
 @click.pass_context
 @click.argument("tool", type=str, required=False, metavar="<tool> or <tool/subtool>")
 @click.option("-d", "--dir", type=click.Path(exists=True), default=".", metavar="<nf-core/modules directory>")
@@ -710,9 +734,9 @@ def bump_version(new_version, dir, nextflow):
 @click.option("-d", "--dir", type=click.Path(exists=True), default=".", help="Pipeline directory. Defaults to CWD")
 @click.option("-b", "--from-branch", type=str, help="The git branch to use to fetch workflow vars.")
 @click.option("-p", "--pull-request", is_flag=True, default=False, help="Make a GitHub pull-request with the changes.")
-@click.option("-r", "--repository", type=str, help="GitHub PR: target repository.")
+@click.option("-g", "--github-repository", type=str, help="GitHub PR: target repository.")
 @click.option("-u", "--username", type=str, help="GitHub PR: auth username.")
-def sync(dir, from_branch, pull_request, repository, username):
+def sync(dir, from_branch, pull_request, github_repository, username):
     """
     Sync a pipeline TEMPLATE branch with the nf-core template.
 
@@ -732,7 +756,7 @@ def sync(dir, from_branch, pull_request, repository, username):
         raise
 
     # Sync the given pipeline dir
-    sync_obj = nf_core.sync.PipelineSync(dir, from_branch, pull_request, repository, username)
+    sync_obj = nf_core.sync.PipelineSync(dir, from_branch, pull_request, github_repository, username)
     try:
         sync_obj.sync()
     except (nf_core.sync.SyncException, nf_core.sync.PullRequestException) as e: