Skip to content

v1.13 - Copper Crocodile
Compare
Choose a tag to compare
@ewels ewels released this 18 Mar 12:03
1.13
03ab820
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

When we do releases, we sometimes like to go big. This is one of those times.

This is a truly huge release of nf-core/tools with a massive amount of new functionality. Please be sure to read the changelog and updated readme carefully!

Note to developers - please run and push the results of nf-core lint --fix before attempting to merge the automated synchronisation PR. This should reduce the size of the diff and make merging a lot easier.

The template and associated updates in this release is for Nextflow DSL1 pipelines. This is likely to be one of the last releases before we officially switch to DSL2. For those of you already with DSL2 pipelines you may need to treat the automated merge PR very carefully. Ask for help on Slack if in any doubt.

Template

  • Major new feature - Validation of pipeline parameters [#426]
    • The addition runs as soon as the pipeline launches and checks the pipeline input parameters two main things:
      • No parameters are supplied that share a name with core Nextflow options (eg. --resume instead of -resume)
      • Supplied parameters validate against the pipeline JSON schema (eg. correct variable types, required values)
    • If either parameter validation fails or the pipeline has errors, a warning is given about any unexpected parameters found which are not described in the pipeline schema.
    • This behaviour can be disabled by using --validate_params false
  • Added profiles to support the Charliecloud and Shifter container engines [#824]
    • Note that Charliecloud requires Nextflow version v21.03.0-edge or later.
  • Profiles for container engines now explicitly disable all other engines [#867]
  • Fixed typo in nf-core-lint CI that prevented the markdown summary from being automatically posted on PRs as a comment.
  • Changed default for --input from data/*{1,2}.fastq.gz to null, as this is now validated by the schema as a required value.
  • Removed support for --name parameter for custom run names.
    • The same functionality for MultiQC still exists with the core Nextflow -name option.
  • Added to template docs about how to identify process name for resource customisation
  • The parameters --max_memory and --max_time are now validated against a regular expression [#793]
    • Must be written in the format 123.GB / 456.h with any of the prefixes listed in the Nextflow docs
    • Bare numbers no longer allowed, avoiding people from trying to specify GB and actually specifying bytes.
  • Switched from cookiecutter to Jinja2 [#880]
  • Finally dropped the wonderful cookiecutter library that was behind the first pipeline template that led to nf-core [#880]
    • Now rendering templates directly using Jinja, which is what cookiecutter was doing anyway

Modules

Initial addition of a number of new helper commands for working with DSL2 modules:

  • modules list - List available modules
  • modules install - Install a module from nf-core/modules
  • modules remove - Remove a module from a pipeline
  • modules create - Create a module from the template
  • modules create-test-yml - Create the test.yml file for a module with md5 sums, tags, commands and names added
  • modules lint - Check a module against nf-core guidelines

You can read more about each of these commands in the main tools documentation (see README.md or https://nf-co.re/tools)

Tools helper code

  • Fixed some bugs in the command line interface for nf-core launch and improved formatting [#829]
  • New functionality for nf-core download to make it compatible with DSL2 pipelines [#832]
    • Singularity images in module files are now discovered and fetched
    • Direct downloads of Singularity images in python allowed (much faster than running singularity pull)
    • Downloads now work with $NXF_SINGULARITY_CACHEDIR so that pipelines sharing containers have efficient downloads
  • Changed behaviour of nf-core sync command [#787]
    • Instead of opening or updating a PR from TEMPLATE directly to dev, a new branch is now created from TEMPLATE and a PR opened from this to dev.
    • This is to make it easier to fix merge conflicts without accidentally bringing the entire pipeline history back into the TEMPLATE branch (which makes subsequent sync merges much more difficult)

Linting

  • Major refactor and rewrite of pipieline linting code
    • Much better code organisation and maintainability
    • New automatically generated documentation using Sphinx
    • Numerous new tests and functions, removal of some unnecessary tests
  • Added lint check for merge markers [#321]
  • Added new option --fix to automatically correct some problems detected by linting
  • Added validation of default params to nf-core schema lint [#823]
  • Added schema validation of GitHub action workflows to lint function [#795]
  • Fixed bug in schema title and description validation
  • Added second progress bar for conda dependencies lint check, as it can be slow [#299]
  • Added new lint test to check files that should be unchanged from the pipeline.
  • Added the possibility to ignore lint tests using a nf-core-lint.yml config file [#809]
Assets 2
Loading