v1.13.3 - Copper Crocodile Resurrection 🐊 🚑 - [2021-03-24]

This is a patch release fixing some nasty bugs that crept into the code base for the nf-core helper tools. There are no changes to the nf-core pipeline template, so there should be no automated pipeline synchronisation pull-requests for this release.

New features

• Running tests twice with nf-core modules create-test-yml to catch unreproducible md5 sums [#890]
• nf-core modules create command - if no bioconda package is found, prompt user for a different bioconda package name
• Updated module template main.nf with new test data paths

Bug fixes

• Fix sync error again where the Nextflow edge release needs to be used for some pipelines
• Fix bug with nf-core lint --release (NameError: name 'os' is not defined)
• Added linebreak to linting comment so that markdown header renders on PR comment properly

v1.13.2 - Copper Crocodile CPR 🐊 🤕

• Added better logging message if a user doesn't specificy the directory correctly with nf-core modules commands [#942]
• Fixed parameter validation bug caused by JSONObject [#937]
• Fixed template creation error regarding file permissions [#932]
• Split the create-lint-wf tests up into separate steps in GitHub Actions to make the CI results easier to read
• Added automated PR comments to the Markdown, YAML and Python lint CI tests to explain failures (tools and pipeline template)
• Make nf-core lint summary table borders coloured according to overall pass / fail status
• Attempted a fix for the automated sync when we submit too many PRs at once [#911]

v1.13.1 - Copper Crocodile Patch 🐊 🏴‍☠️

Minor patch release to scoop up a small few errors discovered after the release.

• Fixed bug in pipeline linting markdown output that gets posted to PR comments [#914]
• Made text for the PR branch CI check less verbose with a TLDR in bold at the top
• A number of minor tweaks to the new nf-core modules lint code

v1.13 - Copper Crocodile

When we do releases, we sometimes like to go big. This is one of those times.

This is a truly huge release of nf-core/tools with a massive amount of new functionality. Please be sure to read the changelog and updated readme carefully!

Note to developers - please run and push the results of nf-core lint --fix before attempting to merge the automated synchronisation PR. This should reduce the size of the diff and make merging a lot easier.

The template and associated updates in this release is for Nextflow DSL1 pipelines. This is likely to be one of the last releases before we officially switch to DSL2. For those of you already with DSL2 pipelines you may need to treat the automated merge PR very carefully. Ask for help on Slack if in any doubt.

Template

• Major new feature - Validation of pipeline parameters [#426]
  • The addition runs as soon as the pipeline launches and checks the pipeline input parameters two main things:
    • No parameters are supplied that share a name with core Nextflow options (eg. --resume instead of -resume)
    • Supplied parameters validate against the pipeline JSON schema (eg. correct variable types, required values)
  • If either parameter validation fails or the pipeline has errors, a warning is given about any unexpected parameters found which are not described in the pipeline schema.
  • This behaviour can be disabled by using --validate_params false
• Added profiles to support the Charliecloud and Shifter container engines [#824]
  • Note that Charliecloud requires Nextflow version v21.03.0-edge or later.
• Profiles for container engines now explicitly disable all other engines [#867]
• Fixed typo in nf-core-lint CI that prevented the markdown summary from being automatically posted on PRs as a comment.
• Changed default for --input from data/*{1,2}.fastq.gz to null, as this is now validated by the schema as a required value.
• Removed support for --name parameter for custom run names.
  • The same functionality for MultiQC still exists with the core Nextflow -name option.
• Added to template docs about how to identify process name for resource customisation
• The parameters --max_memory and --max_time are now validated against a regular expression [#793]
  • Must be written in the format 123.GB / 456.h with any of the prefixes listed in the Nextflow docs
  • Bare numbers no longer allowed, avoiding people from trying to specify GB and actually specifying bytes.
• Switched from cookiecutter to Jinja2 [#880]
• Finally dropped the wonderful cookiecutter library that was behind the first pipeline template that led to nf-core [#880]
  • Now rendering templates directly using Jinja, which is what cookiecutter was doing anyway

Modules

Initial addition of a number of new helper commands for working with DSL2 modules:

• modules list - List available modules
• modules install - Install a module from nf-core/modules
• modules remove - Remove a module from a pipeline
• modules create - Create a module from the template
• modules create-test-yml - Create the test.yml file for a module with md5 sums, tags, commands and names added
• modules lint - Check a module against nf-core guidelines

You can read more about each of these commands in the main tools documentation (see README.md or https://nf-co.re/tools)

Tools helper code

• Fixed some bugs in the command line interface for nf-core launch and improved formatting [#829]
• New functionality for nf-core download to make it compatible with DSL2 pipelines [#832]
  • Singularity images in module files are now discovered and fetched
  • Direct downloads of Singularity images in python allowed (much faster than running singularity pull)
  • Downloads now work with $NXF_SINGULARITY_CACHEDIR so that pipelines sharing containers have efficient downloads
• Changed behaviour of nf-core sync command [#787]
  • Instead of opening or updating a PR from TEMPLATE directly to dev, a new branch is now created from TEMPLATE and a PR opened from this to dev.
  • This is to make it easier to fix merge conflicts without accidentally bringing the entire pipeline history back into the TEMPLATE branch (which makes subsequent sync merges much more difficult)

Linting

• Major refactor and rewrite of pipieline linting code
  • Much better code organisation and maintainability
  • New automatically generated documentation using Sphinx
  • Numerous new tests and functions, removal of some unnecessary tests
• Added lint check for merge markers [#321]
• Added new option --fix to automatically correct some problems detected by linting
• Added validation of default params to nf-core schema lint [#823]
• Added schema validation of GitHub action workflows to lint function [#795]
• Fixed bug in schema title and description validation
• Added second progress bar for conda dependencies lint check, as it can be slow [#299]
• Added new lint test to check files that should be unchanged from the pipeline.
• Added the possibility to ignore lint tests using a nf-core-lint.yml config file [#809]

1.12.1 - Silver Dolphin

Minor release, primarily to fix a problem introduced in v1.12 where not all instances of baseDir had been switched with projectDir. However, the elves have been busy and managed to squeeze in a few extra goodies along with the fix:

Template

• Finished switch from $baseDir to $projectDir in iGenomes.conf and main.nf
  • Main fix is for smail_fields which was a bug introduced in the previous release. Sorry about that!
• Ported a number of small content tweaks from nf-core/eager to the template [#786]
  • Better contributing documentation, more placeholders in documentation files, more relaxed markdownlint exceptions for certain HTML tags, more content for the PR and issue templates.

Tools helper code

• Pipeline schema: make parameters of type range to number. [#738]
• Respect $NXF_HOME when looking for pipelines with nf-core list [#798]
• Swapped PyInquirer with questionary for command line questions in launch.py [#726]
  • This should fix conda installation issues that some people had been hitting
  • The change also allows other improvements to the UI
• Fix linting crash when a file deleted but not yet staged in git [#796]

1.12 - Mercury Weasel

A bit of a fast cycle on this release, it is needed urgently because GitHub Actions CI tests will be failing on all pipelines due to the set-env and add-path commands being depreciated (see the GitHub blog post for details).

Tools helper code

• Updated nf_core documentation generator for building https://nf-co.re/tools-docs/

Template

• Make CI comments work with PRs from forks [#765]
  • Branch protection and linting results should now show on all PRs
• Updated GitHub issue templates, which had stopped working
• Refactored GitHub Actions so that the AWS full-scale tests are triggered after docker build is finished
  • DockerHub push workflow split into two - one for dev, one for releases
• Updated actions to no longer use set-env which is now depreciating [#739]
• Added config import for test_full in nextflow.config
• Switched depreciated $baseDir to $projectDir
• Updated minimum Nextflow version to 20.04.10
• Make Nextflow installation less verbose in GitHub Actions [#780]

Linting

• Updated code to display colours in GitHub Actions log output
• Allow tests to pass with dev version of nf-core/tools (previous failure due to base image version)
• Lint code no longer tries to post GitHub PR comments. This is now done in a GitHub Action only.

1.11 - Iron Tiger

Template

• Fix command error in awstest.yml GitHub Action workflow.
• Allow manual triggering of AWS test GitHub Action workflows.
• Remove TODO item, which was proposing the usage of additional files beside usage.md and output.md for documentation.
• Added a Podman profile, which enables Podman as container.
• Updated linting for GitHub actions AWS tests workflows.

Linting

• Made a base-level Dockerfile a warning instead of failure
• Added a lint failure if the old bin/markdown_to_html.r script is found
• Update rich package dependency and use new markup escaping to change [[!]] back to [!] again

Other

• Pipeline sync - fetch full repo when checking out before sync
• Sync - Add GitHub actions manual trigger option

1.10.2 - Copper Camel [brought back from the dead]

Second patch release to address some small errors discovered in the pipeline template.
Apologies for the inconvenience.

• Fix syntax error in /push_dockerhub.yml GitHub Action workflow
• Change params.readPaths -> params.input_paths in test_full.config
• Check results when posting the lint results as a GitHub comment
  • This feature is unfortunately not possible when making PRs from forks outside of the nf-core organisation for now.
• More major refactoring of the automated pipeline sync
  • New GitHub Actions matrix parallelisation of sync jobs across pipelines [#673]
  • Removed the --all behaviour from nf-core sync as we no longer need it
  • Sync now uses a new list of pipelines on the website which does not include archived pipelines [#712]
  • When making a PR it checks if a PR already exists - if so it updates it [#710]
  • More tests and code refactoring for more stable code. Hopefully fixes 404 error [#711]

1.10.1 - Copper Camel [patch]

Patch release to fix the automatic template synchronisation, which failed in the v1.10 release.

• Improved logging: nf-core --log-file log.txt now saves a verbose log to disk.
• nf-core/tools GitHub Actions pipeline sync now uploads verbose log as an artifact.
• Sync - fixed several minor bugs, made logging less verbose.
• Python Rich library updated to >=4.2.1
• Hopefully fix git config for pipeline sync so that commit comes from @nf-core-bot
• Fix sync auto-PR text indentation so that it doesn't all show as code
• Added explicit flag --show-passed for nf-core lint instead of taking logging verbosity

1.10 - Copper Camel

Pipeline schema

This release of nf-core/tools introduces a major change / new feature: pipeline schema.
These are JSON Schema files that describe all of the parameters for a given
pipeline with their ID, a description, a longer help text, an optional default value, a variable type
(eg. string or boolean) and more.

The files will be used in a number of places:

• Automatic validation of supplied parameters when running pipelines
  • Pipeline execution can be immediately stopped if a required param is missing,
    or does not conform to the patterns / allowed values in the schema.
• Generation of pipeline command-line help
  • Running nextflow run <pipeline> --help will use the schema to generate a help text automatically
• Building online documentation on the nf-core website
• Integration with 3rd party graphical user interfaces

To support these new schema files, nf-core/tools now comes with a new set of commands: nf-core schema.

• Pipeline schema can be generated or updated using nf-core schema build - this takes the parameters from
  the pipeline config file and prompts the developer for any mismatch between schema and pipeline.
  • Once a skeleton Schema file has been built, the command makes use of a new nf-core website tool to provide
    a user friendly graphical interface for developers to add content to their schema: https://nf-co.re/pipeline_schema_builder
• Pipelines will be automatically tested for valid schema that describe all pipeline parameters using the
  nf-core schema lint command (also included as part of the main nf-core lint command).
• Users can validate their set of pipeline inputs using the nf-core schema validate command.

In addition to the new schema commands, the nf-core launch command has been completely rewritten from
scratch to make use of the new pipeline schema. This command can use either an interactive command-line
prompt or a rich web interface to help users set parameters for a pipeline run.

The parameter descriptions and help text are fully used and embedded into the launch interfaces to make
this process as user-friendly as possible. We hope that it's particularly well suited to those new to nf-core.

Whilst we appreciate that this new feature will add a little work for pipeline developers, we're excited at
the possibilities that it brings. If you have any feedback or suggestions, please let us know either here on
GitHub or on the nf-core #json-schema Slack channel.

Python code formatting

We have adopted the use of the Black Python code formatter.
This ensures a harmonised code formatting style throughout the package, from all contributors.
If you are editing any Python code in nf-core/tools you must now pass the files through Black when
making a pull-request. See .github/CONTRIBUTING.md for details.

Template

• Add --publish_dir_mode parameter #585
• Isolate R library paths to those in container #541
• Added new style of pipeline parameters JSON schema to pipeline template
• Add ability to attach MultiQC reports to completion emails when using mail
• Update output.md and add in 'Pipeline information' section describing standard NF and pipeline reporting.
• Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub)
• Add Slack channel badge in pipeline README
• Allow multiple container tags in ci.yml if performing multiple tests in parallel
• Add AWS CI tests and full tests GitHub Actions workflows
• Update AWS CI tests and full tests secrets names
• Added macs_gsize for danRer10, based on this post
• Add information about config files used for workflow execution (workflow.configFiles) to summary
• Fix markdown_to_html.py to work with Python 2 and 3.
• Change params.reads -> params.input
• Change params.readPaths -> params.input_paths
• Added a .github/.dockstore.yml config file for automatic workflow registration with dockstore.org

Linting

• Refactored PR branch tests to be a little clearer.
• Linting error docs explain how to add an additional branch protecton rule to the branch.yml GitHub Actions workflow.
• Adapted linting docs to the new PR branch tests.
• Failure for missing the readme bioconda badge is now a warn, in case this badge is not relevant
• Added test for template {{ cookiecutter.var }} placeholders
• Fix failure when providing version along with build id for Conda packages
• New --json and --markdown options to print lint results to JSON / markdown files
• Linting code now automatically posts warning / failing results to GitHub PRs as a comment if it can
• Added AWS GitHub Actions workflows linting
• Fail if params.input isn't defined.
• Beautiful new progress bar to look at whilst linting is running and awesome new formatted output on the command line 😍
  • All made using the excellent rich python library - check it out!
• Tests looking for TODO strings should now ignore editor backup files. #477

nf-core/tools Continuous Integration

• Added CI test to check for PRs against master in tools repo
• CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong
• Move some of the issue and PR templates into HTML <!-- comments --> so that they don't show in issues / PRs

Other

• Describe alternative installation method via conda with conda env create
• nf-core/tools version number now printed underneath header artwork
• Bumped Conda version shipped with nfcore/base to 4.8.2
• Added log message when creating new pipelines that people should talk to the community about their plans
• Fixed 'on completion' emails sent using the mail command not containing body text.
• Improved command-line help text for nf-core/tools
• nf-core list now hides archived pipelines unless --show_archived flag is set
• Command line tools now checks if there is a new version of nf-core/tools available
  • Disable this by setting the environment variable NFCORE_NO_VERSION_CHECK, eg. export NFCORE_NO_VERSION_CHECK=1
• Better command-line output formatting of nearly all nf-core commands using rich