From 3e0895e5efb8f14266d3bdb18cad56766658e148 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 19 Nov 2020 13:57:35 +0100 Subject: [PATCH 01/50] Version bump to v1.13dev --- CHANGELOG.md | 10 ++++++++++ setup.py | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e949ddce9d..13192a2bff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # nf-core/tools: Changelog +## v1.13dev + +### Tools helper code + +### Template + +### Linting + +### Other + ## [v1.12 - Mercury Weasel](https://github.com/nf-core/tools/releases/tag/1.12) - [2020-11-19] ### Tools helper code diff --git a/setup.py b/setup.py index de3c457207..233f5c3216 100644 --- a/setup.py +++ b/setup.py @@ -3,7 +3,7 @@ from setuptools import setup, find_packages import sys -version = "1.12" +version = "1.13dev" with open("README.md") as f: readme = f.read() From 348b34e7ed87ae3192e4e2ed50ffc0fb42b700bd Mon Sep 17 00:00:00 2001 From: James Fellows Yates Date: Fri, 20 Nov 2020 10:27:47 +0100 Subject: [PATCH 02/50] Add extra sections and content prompts --- .../{{cookiecutter.name_noslash}}/README.md | 22 ++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index 6b8859a531..2f0a26df62 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -12,6 +12,8 @@ ## Introduction +**{{ cookiecutter.name }}** is a bioinformatics best-practise analysis pipeline for + The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It comes with docker containers making installation trivial and results highly reproducible. ## Quick Start @@ -38,6 +40,15 @@ The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool See [usage docs](https://nf-co.re/{{ cookiecutter.short_name }}/usage) for all of the available options when running the pipeline. +## Pipeline Summary + +By default, the pipeline currently performs the following: + + + +* Sequencing quality control (`FastQC`) +* Overall pipeline run summaries (`MultiQC`) + ## Documentation The {{ cookiecutter.name }} pipeline comes with documentation about the pipeline: [usage](https://nf-co.re/{{ cookiecutter.short_name }}/usage) and [output](https://nf-co.re/{{ cookiecutter.short_name }}/output). @@ -48,13 +59,18 @@ The {{ cookiecutter.name }} pipeline comes with documentation about the pipeline {{ cookiecutter.name }} was originally written by {{ cookiecutter.author }}. +We thank the following people for their extensive assistance in the development +of this pipeline: + + + ## Contributions and Support If you would like to contribute to this pipeline, please see the [contributing guidelines](.github/CONTRIBUTING.md). For further information or help, don't hesitate to get in touch on the [Slack `#{{ cookiecutter.short_name }}` channel](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) (you can join with [this invite](https://nf-co.re/join/slack)). -## Citation +## Citations @@ -67,3 +83,7 @@ You can cite the `nf-core` publication as follows: > > _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). > ReadCube: [Full Access Link](https://rdcu.be/b1GjZ) + +In addition, references of tools and data used in this pipeline are as follows: + + \ No newline at end of file From 2d617b57272b0c6a444fe1e47653dfb8ce1533f1 Mon Sep 17 00:00:00 2001 From: James Fellows Yates Date: Fri, 20 Nov 2020 10:47:18 +0100 Subject: [PATCH 03/50] Extended contribution guidelines - conventions require more discussion --- .../.github/CONTRIBUTING.md | 77 ++++++++++++++++++- 1 file changed, 74 insertions(+), 3 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index 8ab3b9bd2e..dd33cb54dd 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -18,8 +18,9 @@ If you'd like to write some code for {{ cookiecutter.name }}, the standard workf 1. Check that there isn't already an issue about your idea in the [{{ cookiecutter.name }} issues](https://github.com/{{ cookiecutter.name }}/issues) to avoid duplicating work * If there isn't one already, please create one so that others know you're working on this 2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [{{ cookiecutter.name }} repository](https://github.com/{{ cookiecutter.name }}) to your GitHub account -3. Make the necessary changes / additions within your forked repository -4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged +3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions) +4. Use `nf-core schema build .` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10). +5. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged If you're not used to this workflow with git, you can start with some [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests) or even their [excellent `git` resources](https://try.github.io/). @@ -52,6 +53,76 @@ These tests are run both with the latest available version of `Nextflow` and als * Fix the bug, and bump version (X.Y.Z+1). * A PR should be made on `master` from patch to directly this particular bug. -## Getting help +## Getting Help For further information/help, please consult the [{{ cookiecutter.name }} documentation](https://nf-co.re/{{ cookiecutter.short_name }}/usage) and don't hesitate to get in touch on the nf-core Slack [#{{ cookiecutter.short_name }}](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) channel ([join our Slack here](https://nf-co.re/join/slack)). + +## Pipeline Contribution Conventions + +To make the {{ cookiecutter.short_name }} code and processing logic more understandable for new contributors and to ensure quality, we semi-standardise the way the code and other contributions are written. + +### Adding a New Module + +If you wish to contribute a new module, please use the following coding standards: + +1. Define the corresponding input channel into your new process from the expected previous process channel +2. Write the process block (see below). +3. Define the output channel if needed (see below). +4. Add any new flags/options to `nextflow.config` with a default (see below). +5. Add any new flags/options to `nextflow_schema.json` with help text (with `nf-core schema build .`) +6. Add any new flags/options to the help message (for integer/text parameters, print to help the corresponding `nextflow.config` parameter). +7. Add sanity checks for all relevant parameters. +8. Add any new software to the `scrape_software_versions.py` script in `bin/` and the version command to the `scrape_software_versions` process in `main.nf`. +9. Do local tests that the new code works properly and as expected. +10. Add a new test command in `.github/workflow/ci.yaml`. +11. If applicable add a [MultiQC](https://https://multiqc.info/) module. +12. Update MultiQC config `assets/multiqc_config.yaml` so relevant suffixes, name clean up, General Statistics Table column order, and module figures are in the right order. +13. Optional: Add any descriptions of MultiQC report sections and output files to `docs/output.md`. + +More details are as follow + +### Default Values + +Default values should go in `nextflow.config` under the `params` scope, and `nextflow_schema.json` (latter with `nf-core schema build .`) + +### Default Processes Resource Requirements + +Defining recommended 'minimum' resource requirements (CPUs/Memory) for a process should be defined in `conf/base.config`. This can be utilised within the process using `${task.cpu}` or `${task.memory}` variables in the `script:` block. + +### Naming Schemes + +Please use the following naming schemes, to make it easy to understand what is going where. + +* initial process channel: `ch_output_from_` +* intermediate and terminal channels: `ch__for_` + +### Nextflow Version Bumping + +If you have agreement from reviewers, you may bump the 'default' minimum version of nextflow (e.g. for testing), with `nf-core bump-version`. + +### Software Version Reporting + +If you add a new tool to the pipeline, please ensure you add the information of the tool to the `get_software_version` process. + +Add to the script block of the process, something like the following: + +```bash + --version &> v_.txt 2>&1 || true +``` + +or + +```bash + --help | head -n 1 &> v_.txt 2>&1 || true +``` + +You then need to edit the script `bin/scrape_software_versions.py` to + +1. add a (python) regex for your tools --version output (as in stored in the `v_.txt` file), to ensure the version is reported as a `v` and the version number e.g. `v2.1.1` +2. add a HTML block entry to the `OrderedDict` for formatting in MultiQC. + +> If a tool does not unfortunately offer any printing of version data, you may add this 'manually' e.g. with `echo "v1.1" > v_.txt` + +### Images and Figures + +For overview images and other documents we follow the nf-core [style guidelines and examples](https://nf-co.re/developers/design_guidelines). From e5d16af1edc50afca8c3e8d227f29e4cc31714ea Mon Sep 17 00:00:00 2001 From: James Fellows Yates Date: Fri, 20 Nov 2020 10:47:32 +0100 Subject: [PATCH 04/50] Allow some inline HTML in markdown lint --- .../{{cookiecutter.name_noslash}}/.github/markdownlint.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/markdownlint.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/markdownlint.yml index 96b12a7039..0967bbbb81 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/markdownlint.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/markdownlint.yml @@ -3,3 +3,10 @@ default: true, line-length: false no-duplicate-header: siblings_only: true +no-inline-html: + allowed_elements: + - img + - p + - kbd + - details + - summary From a3c6f4be644cb3923ea428e2a801bfd6a85a001c Mon Sep 17 00:00:00 2001 From: James Fellows Yates Date: Fri, 20 Nov 2020 10:53:22 +0100 Subject: [PATCH 05/50] Update bug report to include checklists --- .../.github/ISSUE_TEMPLATE/bug_report.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md index 6f384d628a..560fd365cf 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md @@ -13,6 +13,16 @@ Thanks for telling us about a problem with the pipeline. Please delete this text and anything that's not relevant from the template below: --> +## Check Documentation + +Have you checked in the following places for your error?: + +- [ ] [nf-co.re troubleshooting(https://nf-co.re/usage/troubleshooting) +- [ ] [{{ cookiecutter.name }}-specific documentation](https://nf-co.re/{{ cookiecutter.name }}/usage) + +Please also check the the corresponding version's documentation, if not +testing the latest release. + ## Description of the bug @@ -28,6 +38,14 @@ Steps to reproduce the behaviour: +## Log files + +Have you provided the following extra information/files: + +- [] The command used to run the pipeline +- [] The `.nextflow.log` file (which is a hidden file in whichever place you _ran_ + the pipeline from - not necessarily in the output directory!) + ## System - Hardware: From d68e8a2af7d3353d5ca71f64f82659db9d731cf4 Mon Sep 17 00:00:00 2001 From: James Fellows Yates Date: Fri, 20 Nov 2020 10:58:31 +0100 Subject: [PATCH 06/50] Addition checks to PR template and checklist fix --- .../.github/ISSUE_TEMPLATE/bug_report.md | 4 ++-- .../.github/PULL_REQUEST_TEMPLATE.md | 16 ++++++++++++---- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md index 560fd365cf..5f67bb76c7 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md @@ -42,8 +42,8 @@ Steps to reproduce the behaviour: Have you provided the following extra information/files: -- [] The command used to run the pipeline -- [] The `.nextflow.log` file (which is a hidden file in whichever place you _ran_ +- [ ] The command used to run the pipeline +- [ ] The `.nextflow.log` file (which is a hidden file in whichever place you _ran_ the pipeline from - not necessarily in the output directory!) ## System diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md index 25f24d6c1f..eff042c5a2 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md @@ -13,8 +13,16 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/{{ cookiecut ## PR checklist -- [ ] This comment contains a description of changes (with reason) -- [ ] `CHANGELOG.md` is updated +- [ ] This comment contains a description of changes (with reason). - [ ] If you've fixed a bug or added code that should be tested, add tests! -- [ ] Documentation in `docs` is updated -- [ ] If necessary, also make a PR on the [{{ cookiecutter.name }} branch on the nf-core/test-datasets repo](https://github.com/nf-core/test-datasets/pull/new/{{ cookiecutter.name }}) + - [ ] If you've added a new tool - add to the software_versions process and a regex to `scrape_software_versions.py` + - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/{{ cookiecutter.name }}/tree/master/.github/CONTRIBUTING.md) + - [ ] If necessary, also make a PR on the {{ cookiecutter.name }} _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. +- [ ] Make sure your code lints (`nf-core lint .`). +- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`). +- [ ] Usage Documentation in `docs/usage.md` is updated. +- [ ] Output Documentation in `docs/output.md` is updated. +- [ ] `CHANGELOG.md` is updated. +- [ ] `README.md` is updated (including new tool citations and authors/contributors). + +**Learn more about contributing:** https://github.com/nf-core/eager/tree/master/.github/CONTRIBUTING.md From 88c85d57e6da07440f6d99c036628e624985c6d3 Mon Sep 17 00:00:00 2001 From: Julianus Pfeuffer Date: Mon, 23 Nov 2020 10:10:08 +0100 Subject: [PATCH 07/50] Remove remaining baseDir occurrance --- nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index f5c767e31e..e8f861f054 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -336,7 +336,7 @@ workflow.onComplete { def email_html = html_template.toString() // Render the sendmail template - def smail_fields = [ email: email_address, subject: subject, email_txt: email_txt, email_html: email_html, baseDir: "$projectDir", mqcFile: mqc_report, mqcMaxSize: params.max_multiqc_email_size.toBytes() ] + def smail_fields = [ email: email_address, subject: subject, email_txt: email_txt, email_html: email_html, projectDir: "$projectDir", mqcFile: mqc_report, mqcMaxSize: params.max_multiqc_email_size.toBytes() ] def sf = new File("$projectDir/assets/sendmail_template.txt") def sendmail_template = engine.createTemplate(sf).make(smail_fields) def sendmail_html = sendmail_template.toString() From d510b157bdfabb2a32515403a7180ff6dd59bb21 Mon Sep 17 00:00:00 2001 From: Julianus Pfeuffer Date: Tue, 24 Nov 2020 17:40:21 +0100 Subject: [PATCH 08/50] baseDir->projectDir in igenomes.conf --- .../conf/igenomes.config | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config index caeafceb25..31b7ee6130 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config @@ -21,7 +21,7 @@ params { readme = "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Annotation/README.txt" mito_name = "MT" macs_gsize = "2.7e9" - blacklist = "${baseDir}/assets/blacklists/GRCh37-blacklist.bed" + blacklist = "${projectDir}/assets/blacklists/GRCh37-blacklist.bed" } 'GRCh38' { fasta = "${params.igenomes_base}/Homo_sapiens/NCBI/GRCh38/Sequence/WholeGenomeFasta/genome.fa" @@ -33,7 +33,7 @@ params { bed12 = "${params.igenomes_base}/Homo_sapiens/NCBI/GRCh38/Annotation/Genes/genes.bed" mito_name = "chrM" macs_gsize = "2.7e9" - blacklist = "${baseDir}/assets/blacklists/hg38-blacklist.bed" + blacklist = "${projectDir}/assets/blacklists/hg38-blacklist.bed" } 'GRCm38' { fasta = "${params.igenomes_base}/Mus_musculus/Ensembl/GRCm38/Sequence/WholeGenomeFasta/genome.fa" @@ -46,7 +46,7 @@ params { readme = "${params.igenomes_base}/Mus_musculus/Ensembl/GRCm38/Annotation/README.txt" mito_name = "MT" macs_gsize = "1.87e9" - blacklist = "${baseDir}/assets/blacklists/GRCm38-blacklist.bed" + blacklist = "${projectDir}/assets/blacklists/GRCm38-blacklist.bed" } 'TAIR10' { fasta = "${params.igenomes_base}/Arabidopsis_thaliana/Ensembl/TAIR10/Sequence/WholeGenomeFasta/genome.fa" @@ -270,7 +270,7 @@ params { bed12 = "${params.igenomes_base}/Homo_sapiens/UCSC/hg38/Annotation/Genes/genes.bed" mito_name = "chrM" macs_gsize = "2.7e9" - blacklist = "${baseDir}/assets/blacklists/hg38-blacklist.bed" + blacklist = "${projectDir}/assets/blacklists/hg38-blacklist.bed" } 'hg19' { fasta = "${params.igenomes_base}/Homo_sapiens/UCSC/hg19/Sequence/WholeGenomeFasta/genome.fa" @@ -283,7 +283,7 @@ params { readme = "${params.igenomes_base}/Homo_sapiens/UCSC/hg19/Annotation/README.txt" mito_name = "chrM" macs_gsize = "2.7e9" - blacklist = "${baseDir}/assets/blacklists/hg19-blacklist.bed" + blacklist = "${projectDir}/assets/blacklists/hg19-blacklist.bed" } 'mm10' { fasta = "${params.igenomes_base}/Mus_musculus/UCSC/mm10/Sequence/WholeGenomeFasta/genome.fa" @@ -296,7 +296,7 @@ params { readme = "${params.igenomes_base}/Mus_musculus/UCSC/mm10/Annotation/README.txt" mito_name = "chrM" macs_gsize = "1.87e9" - blacklist = "${baseDir}/assets/blacklists/mm10-blacklist.bed" + blacklist = "${projectDir}/assets/blacklists/mm10-blacklist.bed" } 'bosTau8' { fasta = "${params.igenomes_base}/Bos_taurus/UCSC/bosTau8/Sequence/WholeGenomeFasta/genome.fa" From 378a688e18c7d5f7d2fa66c25772101aebe468d6 Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Thu, 26 Nov 2020 12:53:10 +0100 Subject: [PATCH 09/50] Swapped pyinquirer with questionary --- nf_core/launch.py | 51 +++++++++++++++++++---------------------------- setup.py | 2 +- 2 files changed, 21 insertions(+), 32 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 8d64a612b3..630d87e997 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -10,7 +10,7 @@ import json import logging import os -import PyInquirer +import questionary import re import subprocess import textwrap @@ -20,16 +20,6 @@ log = logging.getLogger(__name__) -# -# NOTE: When PyInquirer 1.0.3 is released we can capture keyboard interruptions -# in a nicer way # with the raise_keyboard_interrupt=True argument in the PyInquirer.prompt() calls -# It also allows list selections to have a default set. -# -# Until then we have workarounds: -# * Default list item is moved to the top of the list -# * We manually raise a KeyboardInterrupt if we get None back from a question -# - class Launch(object): """ Class to hold config option to launch a pipeline """ @@ -257,8 +247,8 @@ def prompt_web_gui(self): "message": "Choose launch method", "choices": ["Web based", "Command line"], } - answer = PyInquirer.prompt([question]) - # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + answer = questionary.prompt([question]) + # Raise keyboard interrupt if answer == {}: raise KeyboardInterrupt return answer["use_web_gui"] == "Web based" @@ -347,14 +337,14 @@ def sanitise_web_response(self): The web builder returns everything as strings. Use the functions defined in the cli wizard to convert to the correct types. """ - # Collect pyinquirer objects for each defined input_param - pyinquirer_objects = {} + # Collect questionary objects for each defined input_param + questionare_objects = {} for param_id, param_obj in self.schema_obj.schema.get("properties", {}).items(): - pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False) + questionare_objects[param_id] = self.single_param_to_questionare(param_id, param_obj, print_help=False) for d_key, definition in self.schema_obj.schema.get("definitions", {}).items(): for param_id, param_obj in definition.get("properties", {}).items(): - pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False) + questionare_objects[param_id] = self.single_param_to_questionare(param_id, param_obj, print_help=False) # Go through input params and sanitise for params in [self.nxf_flags, self.schema_obj.input_params]: @@ -364,7 +354,7 @@ def sanitise_web_response(self): del params[param_id] continue # Run filter function on value - filter_func = pyinquirer_objects.get(param_id, {}).get("filter") + filter_func = questionare_objects.get(param_id, {}).get("filter") if filter_func is not None: params[param_id] = filter_func(params[param_id]) @@ -396,17 +386,17 @@ def prompt_param(self, param_id, param_obj, is_required, answers): """Prompt for a single parameter""" # Print the question - question = self.single_param_to_pyinquirer(param_id, param_obj, answers) - answer = PyInquirer.prompt([question]) - # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + question = self.single_param_to_questionare(param_id, param_obj, answers) + answer = questionary.prompt([question]) + # Raise keyboard interrupg if answer == {}: raise KeyboardInterrupt # If required and got an empty reponse, ask again while type(answer[param_id]) is str and answer[param_id].strip() == "" and is_required: log.error("'–-{}' is required".format(param_id)) - answer = PyInquirer.prompt([question]) - # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + answer = questionary.prompt([question]) + # Raise keyboard interrupt if answer == {}: raise KeyboardInterrupt @@ -430,7 +420,7 @@ def prompt_group(self, group_id, group_obj): "type": "list", "name": group_id, "message": group_obj.get("title", group_id), - "choices": ["Continue >>", PyInquirer.Separator()], + "choices": ["Continue >>", questionary.Separator()], } for param_id, param in group_obj["properties"].items(): @@ -445,8 +435,8 @@ def prompt_group(self, group_id, group_obj): answers = {} while not while_break: self.print_param_header(group_id, group_obj) - answer = PyInquirer.prompt([question]) - # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + answer = questionary.prompt([question]) + # Raise keyboard interrupg if answer == {}: raise KeyboardInterrupt if answer[group_id] == "Continue >>": @@ -465,8 +455,8 @@ def prompt_group(self, group_id, group_obj): return answers - def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_help=True): - """Convert a JSONSchema param to a PyInquirer question + def single_param_to_questionare(self, param_id, param_obj, answers=None, print_help=True): + """Convert a JSONSchema param to a Questionary question Args: param_id: Parameter ID (string) @@ -475,7 +465,7 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_he print_help: If description and help_text should be printed (bool) Returns: - Single PyInquirer dict, to be appended to questions list + Single Questionary dict, to be appended to questions list """ if answers is None: answers = {} @@ -620,9 +610,8 @@ def validate_pattern(val): question["validate"] = validate_pattern - # WORKAROUND - PyInquirer <1.0.3 cannot have a default position in a list + # WORKAROUND - Questionary cannot have a default position in a list # For now, move the default option to the top. - # TODO: Delete this code when PyInquirer >=1.0.3 is released. if question["type"] == "list" and "default" in question: try: question["choices"].remove(question["default"]) diff --git a/setup.py b/setup.py index 233f5c3216..4f727720e7 100644 --- a/setup.py +++ b/setup.py @@ -36,7 +36,7 @@ "GitPython", "jinja2", "jsonschema", - "PyInquirer==1.0.2", + "questionary", "pyyaml", "requests", "requests_cache", From b9622622778303d1759c4cc34c50020d375c4922 Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Thu, 26 Nov 2020 13:30:49 +0100 Subject: [PATCH 10/50] fixed test_launch.py, updated changelog --- CHANGELOG.md | 2 +- tests/test_launch.py | 32 ++++++++++++++++---------------- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 13192a2bff..cf50505ae2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,7 @@ # nf-core/tools: Changelog ## v1.13dev - +* swapped PyInquirer with questionary for command line questions in launch.py ### Tools helper code ### Template diff --git a/tests/test_launch.py b/tests/test_launch.py index 70acc982a4..1756a9b7bb 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -93,21 +93,21 @@ def test_nf_merge_schema(self): assert self.launcher.schema_obj.schema["allOf"][0] == {"$ref": "#/definitions/coreNextflow"} assert "-resume" in self.launcher.schema_obj.schema["definitions"]["coreNextflow"]["properties"] - def test_ob_to_pyinquirer_string(self): + def test_ob_to_questionary_string(self): """ Check converting a python dict to a pyenquirer format - simple strings """ sc_obj = { "type": "string", "default": "data/*{1,2}.fastq.gz", } - result = self.launcher.single_param_to_pyinquirer("input", sc_obj) + result = self.launcher.single_param_to_questionary("input", sc_obj) assert result == {"type": "input", "name": "input", "message": "input", "default": "data/*{1,2}.fastq.gz"} - @mock.patch("PyInquirer.prompt", side_effect=[{"use_web_gui": "Web based"}]) + @mock.patch("questionary.prompt", side_effect=[{"use_web_gui": "Web based"}]) def test_prompt_web_gui_true(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == True - @mock.patch("PyInquirer.prompt", side_effect=[{"use_web_gui": "Command line"}]) + @mock.patch("questionary.prompt", side_effect=[{"use_web_gui": "Command line"}]) def test_prompt_web_gui_false(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == False @@ -198,13 +198,13 @@ def test_sanitise_web_response(self): assert self.launcher.schema_obj.input_params["single_end"] == True assert self.launcher.schema_obj.input_params["max_cpus"] == 12 - def test_ob_to_pyinquirer_bool(self): + def test_ob_to_questionary_bool(self): """ Check converting a python dict to a pyenquirer format - booleans """ sc_obj = { "type": "boolean", "default": "True", } - result = self.launcher.single_param_to_pyinquirer("single_end", sc_obj) + result = self.launcher.single_param_to_questionary("single_end", sc_obj) assert result["type"] == "list" assert result["name"] == "single_end" assert result["message"] == "single_end" @@ -218,10 +218,10 @@ def test_ob_to_pyinquirer_bool(self): assert result["filter"]("false") == False assert result["filter"](False) == False - def test_ob_to_pyinquirer_number(self): + def test_ob_to_questionary_number(self): """ Check converting a python dict to a pyenquirer format - with enum """ sc_obj = {"type": "number", "default": 0.1} - result = self.launcher.single_param_to_pyinquirer("min_reps_consensus", sc_obj) + result = self.launcher.single_param_to_questionary("min_reps_consensus", sc_obj) assert result["type"] == "input" assert result["default"] == "0.1" assert result["validate"]("123") is True @@ -232,10 +232,10 @@ def test_ob_to_pyinquirer_number(self): assert result["filter"]("123.456") == float(123.456) assert result["filter"]("") == "" - def test_ob_to_pyinquirer_integer(self): + def test_ob_to_questionary_integer(self): """ Check converting a python dict to a pyenquirer format - with enum """ sc_obj = {"type": "integer", "default": 1} - result = self.launcher.single_param_to_pyinquirer("broad_cutoff", sc_obj) + result = self.launcher.single_param_to_questionary("broad_cutoff", sc_obj) assert result["type"] == "input" assert result["default"] == "1" assert result["validate"]("123") is True @@ -246,10 +246,10 @@ def test_ob_to_pyinquirer_integer(self): assert result["filter"]("123") == int(123) assert result["filter"]("") == "" - def test_ob_to_pyinquirer_range(self): + def test_ob_to_questionary_range(self): """ Check converting a python dict to a pyenquirer format - with enum """ sc_obj = {"type": "range", "minimum": "10", "maximum": "20", "default": 15} - result = self.launcher.single_param_to_pyinquirer("broad_cutoff", sc_obj) + result = self.launcher.single_param_to_questionary("broad_cutoff", sc_obj) assert result["type"] == "input" assert result["default"] == "15" assert result["validate"]("20") is True @@ -260,10 +260,10 @@ def test_ob_to_pyinquirer_range(self): assert result["filter"]("20") == float(20) assert result["filter"]("") == "" - def test_ob_to_pyinquirer_enum(self): + def test_ob_to_questionary_enum(self): """ Check converting a python dict to a pyenquirer format - with enum """ sc_obj = {"type": "string", "default": "copy", "enum": ["symlink", "rellink"]} - result = self.launcher.single_param_to_pyinquirer("publish_dir_mode", sc_obj) + result = self.launcher.single_param_to_questionary("publish_dir_mode", sc_obj) assert result["type"] == "list" assert result["default"] == "copy" assert result["choices"] == ["symlink", "rellink"] @@ -271,10 +271,10 @@ def test_ob_to_pyinquirer_enum(self): assert result["validate"]("") is True assert result["validate"]("not_allowed") == "Must be one of: symlink, rellink" - def test_ob_to_pyinquirer_pattern(self): + def test_ob_to_questionary_pattern(self): """ Check converting a python dict to a pyenquirer format - with pattern """ sc_obj = {"type": "string", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$"} - result = self.launcher.single_param_to_pyinquirer("email", sc_obj) + result = self.launcher.single_param_to_questionary("email", sc_obj) assert result["type"] == "input" assert result["validate"]("test@email.com") is True assert result["validate"]("") is True From 78ed52ab78682dc7d433c737b064c09f8c31111d Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Thu, 26 Nov 2020 13:34:34 +0100 Subject: [PATCH 11/50] fixed typo --- nf_core/launch.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 630d87e997..cfcc1e043a 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -340,11 +340,11 @@ def sanitise_web_response(self): # Collect questionary objects for each defined input_param questionare_objects = {} for param_id, param_obj in self.schema_obj.schema.get("properties", {}).items(): - questionare_objects[param_id] = self.single_param_to_questionare(param_id, param_obj, print_help=False) + questionare_objects[param_id] = self.single_param_to_questionary(param_id, param_obj, print_help=False) for d_key, definition in self.schema_obj.schema.get("definitions", {}).items(): for param_id, param_obj in definition.get("properties", {}).items(): - questionare_objects[param_id] = self.single_param_to_questionare(param_id, param_obj, print_help=False) + questionare_objects[param_id] = self.single_param_to_questionary(param_id, param_obj, print_help=False) # Go through input params and sanitise for params in [self.nxf_flags, self.schema_obj.input_params]: @@ -386,7 +386,7 @@ def prompt_param(self, param_id, param_obj, is_required, answers): """Prompt for a single parameter""" # Print the question - question = self.single_param_to_questionare(param_id, param_obj, answers) + question = self.single_param_to_questionary(param_id, param_obj, answers) answer = questionary.prompt([question]) # Raise keyboard interrupg if answer == {}: @@ -455,7 +455,7 @@ def prompt_group(self, group_id, group_obj): return answers - def single_param_to_questionare(self, param_id, param_obj, answers=None, print_help=True): + def single_param_to_questionary(self, param_id, param_obj, answers=None, print_help=True): """Convert a JSONSchema param to a Questionary question Args: From b9116e93a81c3bc03bf095a5318d17ae6186ba69 Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Thu, 26 Nov 2020 16:49:18 +0100 Subject: [PATCH 12/50] added blank lines to changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index cf50505ae2..7afac0ec0f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,9 @@ # nf-core/tools: Changelog ## v1.13dev + * swapped PyInquirer with questionary for command line questions in launch.py + ### Tools helper code ### Template From 479887c62c8b50c4ca35e7d2adfa8fc7ad63fcdf Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 26 Nov 2020 22:10:26 +0100 Subject: [PATCH 13/50] Use NXF_HOME env var for nextflow home directory, if set Closes nf-core/tools#798 --- CHANGELOG.md | 2 ++ nf_core/list.py | 4 ++++ nf_core/utils.py | 7 +++++-- 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 13192a2bff..0ebd55dbe8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ ### Tools helper code +* Respect `$NXF_HOME` when looking for pipelines with `nf-core list` [[#798](https://github.com/nf-core/tools/issues/798)] + ### Template ### Linting diff --git a/nf_core/list.py b/nf_core/list.py index c81d35bd42..3b4b188e66 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -128,6 +128,8 @@ def get_local_nf_workflows(self): # Try to guess the local cache directory (much faster than calling nextflow) if len(os.environ.get("NXF_ASSETS", "")) > 0: nextflow_wfdir = os.environ.get("NXF_ASSETS") + else if len(os.environ.get("NXF_HOME", "")) > 0: + nextflow_wfdir = os.path.join(os.environ.get("NXF_HOME"), "assets") else: nextflow_wfdir = os.path.join(os.getenv("HOME"), ".nextflow", "assets") if os.path.isdir(nextflow_wfdir): @@ -348,6 +350,8 @@ def get_local_nf_workflow_details(self): # Try to guess the local cache directory if len(os.environ.get("NXF_ASSETS", "")) > 0: nf_wfdir = os.path.join(os.environ.get("NXF_ASSETS"), self.full_name) + else if len(os.environ.get("NXF_HOME", "")) > 0: + nf_wfdir = os.path.join(os.environ.get("NXF_HOME"), "assets") else: nf_wfdir = os.path.join(os.getenv("HOME"), ".nextflow", "assets", self.full_name) if os.path.isdir(nf_wfdir): diff --git a/nf_core/utils.py b/nf_core/utils.py index f09c4bd3cb..2e6388db31 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -68,9 +68,12 @@ def fetch_wf_config(wf_path): cache_basedir = None cache_path = None + # Nextflow home directory - use env var if set, or default to ~/.nextflow + nxf_home = os.environ.get("NXF_HOME", os.path.join(os.getenv("HOME"), ".nextflow")) + # Build a cache directory if we can - if os.path.isdir(os.path.join(os.getenv("HOME"), ".nextflow")): - cache_basedir = os.path.join(os.getenv("HOME"), ".nextflow", "nf-core") + if os.path.isdir(nxf_home): + cache_basedir = os.path.join(nxf_home, "nf-core") if not os.path.isdir(cache_basedir): os.mkdir(cache_basedir) From b83ac5d23c16114c69bb2804751f49e149917c9a Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 26 Nov 2020 22:15:16 +0100 Subject: [PATCH 14/50] Too many programming languages --- nf_core/list.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/list.py b/nf_core/list.py index 3b4b188e66..45cbeb5d18 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -128,7 +128,7 @@ def get_local_nf_workflows(self): # Try to guess the local cache directory (much faster than calling nextflow) if len(os.environ.get("NXF_ASSETS", "")) > 0: nextflow_wfdir = os.environ.get("NXF_ASSETS") - else if len(os.environ.get("NXF_HOME", "")) > 0: + elif len(os.environ.get("NXF_HOME", "")) > 0: nextflow_wfdir = os.path.join(os.environ.get("NXF_HOME"), "assets") else: nextflow_wfdir = os.path.join(os.getenv("HOME"), ".nextflow", "assets") @@ -350,7 +350,7 @@ def get_local_nf_workflow_details(self): # Try to guess the local cache directory if len(os.environ.get("NXF_ASSETS", "")) > 0: nf_wfdir = os.path.join(os.environ.get("NXF_ASSETS"), self.full_name) - else if len(os.environ.get("NXF_HOME", "")) > 0: + elif len(os.environ.get("NXF_HOME", "")) > 0: nf_wfdir = os.path.join(os.environ.get("NXF_HOME"), "assets") else: nf_wfdir = os.path.join(os.getenv("HOME"), ".nextflow", "assets", self.full_name) From 7919bd25f33f841f08ead72de408b8ad4abcfa80 Mon Sep 17 00:00:00 2001 From: Kevin Menden Date: Fri, 27 Nov 2020 08:43:57 +0100 Subject: [PATCH 15/50] typo fixes Co-authored-by: Phil Ewels --- nf_core/launch.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index cfcc1e043a..0e853843c7 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -338,7 +338,7 @@ def sanitise_web_response(self): Use the functions defined in the cli wizard to convert to the correct types. """ # Collect questionary objects for each defined input_param - questionare_objects = {} + questionary_objects = {} for param_id, param_obj in self.schema_obj.schema.get("properties", {}).items(): questionare_objects[param_id] = self.single_param_to_questionary(param_id, param_obj, print_help=False) @@ -388,7 +388,7 @@ def prompt_param(self, param_id, param_obj, is_required, answers): # Print the question question = self.single_param_to_questionary(param_id, param_obj, answers) answer = questionary.prompt([question]) - # Raise keyboard interrupg + # Raise KeyboardInterrupt exception if answer == {}: raise KeyboardInterrupt @@ -436,7 +436,7 @@ def prompt_group(self, group_id, group_obj): while not while_break: self.print_param_header(group_id, group_obj) answer = questionary.prompt([question]) - # Raise keyboard interrupg + # Raise KeyboardInterrupt exception if answer == {}: raise KeyboardInterrupt if answer[group_id] == "Continue >>": From c97d0661bd2d23ef43a49ecbb79c1b6f886a98e7 Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Fri, 27 Nov 2020 08:55:13 +0100 Subject: [PATCH 16/50] fixed bug with new questionare_objects name --- nf_core/launch.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 0e853843c7..3b21366e42 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -340,11 +340,11 @@ def sanitise_web_response(self): # Collect questionary objects for each defined input_param questionary_objects = {} for param_id, param_obj in self.schema_obj.schema.get("properties", {}).items(): - questionare_objects[param_id] = self.single_param_to_questionary(param_id, param_obj, print_help=False) + questionary_objects[param_id] = self.single_param_to_questionary(param_id, param_obj, print_help=False) for d_key, definition in self.schema_obj.schema.get("definitions", {}).items(): for param_id, param_obj in definition.get("properties", {}).items(): - questionare_objects[param_id] = self.single_param_to_questionary(param_id, param_obj, print_help=False) + questionary_objects[param_id] = self.single_param_to_questionary(param_id, param_obj, print_help=False) # Go through input params and sanitise for params in [self.nxf_flags, self.schema_obj.input_params]: @@ -354,7 +354,7 @@ def sanitise_web_response(self): del params[param_id] continue # Run filter function on value - filter_func = questionare_objects.get(param_id, {}).get("filter") + filter_func = questionary_objects.get(param_id, {}).get("filter") if filter_func is not None: params[param_id] = filter_func(params[param_id]) From e6f87154631ba5ec6a05b0007224a5a261eac647 Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Fri, 27 Nov 2020 10:44:10 +0100 Subject: [PATCH 17/50] catch keyboard interrupt, default list choice --- .gitignore | 5 ++++ nf_core/launch.py | 68 +++++++++++++++++++++-------------------------- 2 files changed, 35 insertions(+), 38 deletions(-) diff --git a/.gitignore b/.gitignore index 84ddfd3a08..1d244caa14 100644 --- a/.gitignore +++ b/.gitignore @@ -112,3 +112,8 @@ ENV/ # Jetbrains IDEs .idea +pip-wheel-metadata/nf_core.dist-info/top_level.txt +pip-wheel-metadata/nf_core.dist-info/METADATA +pip-wheel-metadata/nf_core.dist-info/LICENSE +pip-wheel-metadata/nf_core.dist-info/entry_points.txt +.vscode/settings.json diff --git a/nf_core/launch.py b/nf_core/launch.py index 3b21366e42..116f60d595 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -15,11 +15,26 @@ import subprocess import textwrap import webbrowser +from prompt_toolkit.styles import Style import nf_core.schema, nf_core.utils log = logging.getLogger(__name__) +# Custom style for questionary +nfcore_question_style = Style([ + ('qmark', 'fg:#673ab7 bold'), # token in front of the question + ('question', 'bold'), # question text + ('answer', 'fg:#f44336 bold'), # submitted answer text behind the question + ('pointer', 'fg:#eff53b bold'), # pointer used in select and checkbox prompts + ('highlighted', 'fg:#673ab7 bold'), # pointed-at choice in select and checkbox prompts + ('selected', 'fg:#cc5454'), # style for a selected item of a checkbox + ('separator', 'fg:#cc5454'), # separator in lists + ('instruction', ''), # user instructions for select, rawselect, checkbox + ('text', ''), # plain text + ('disabled', 'fg:#858585 italic') # disabled choices for select and checkbox prompts +]) + class Launch(object): """ Class to hold config option to launch a pipeline """ @@ -246,11 +261,9 @@ def prompt_web_gui(self): "name": "use_web_gui", "message": "Choose launch method", "choices": ["Web based", "Command line"], + "default": "Web based" } - answer = questionary.prompt([question]) - # Raise keyboard interrupt - if answer == {}: - raise KeyboardInterrupt + answer = questionary.unsafe_prompt([question], style=nfcore_question_style) return answer["use_web_gui"] == "Web based" def launch_web_gui(self): @@ -387,18 +400,12 @@ def prompt_param(self, param_id, param_obj, is_required, answers): # Print the question question = self.single_param_to_questionary(param_id, param_obj, answers) - answer = questionary.prompt([question]) - # Raise KeyboardInterrupt exception - if answer == {}: - raise KeyboardInterrupt + answer = questionary.unsafe_prompt([question], style=nfcore_question_style) # If required and got an empty reponse, ask again while type(answer[param_id]) is str and answer[param_id].strip() == "" and is_required: log.error("'–-{}' is required".format(param_id)) - answer = questionary.prompt([question]) - # Raise keyboard interrupt - if answer == {}: - raise KeyboardInterrupt + answer = questionary.unsafe_prompt([question], style=nfcore_question_style) # Don't return empty answers if answer[param_id] == "": @@ -435,10 +442,7 @@ def prompt_group(self, group_id, group_obj): answers = {} while not while_break: self.print_param_header(group_id, group_obj) - answer = questionary.prompt([question]) - # Raise KeyboardInterrupt exception - if answer == {}: - raise KeyboardInterrupt + answer = questionary.unsafe_prompt([question], style=nfcore_question_style) if answer[group_id] == "Continue >>": while_break = True # Check if there are any required parameters that don't have answers @@ -588,15 +592,15 @@ def filter_range(val): question["type"] = "list" question["choices"] = param_obj["enum"] - # Validate enum from schema - def validate_enum(val): - if val == "": - return True - if val in param_obj["enum"]: - return True - return "Must be one of: {}".format(", ".join(param_obj["enum"])) + # # Validate enum from schema + # def validate_enum(val): + # if val == "": + # return True + # if val in param_obj["enum"]: + # return True + # return "Must be one of: {}".format(", ".join(param_obj["enum"])) - question["validate"] = validate_enum + #question["validate"] = validate_enum # Validate pattern from schema if "pattern" in param_obj: @@ -610,20 +614,6 @@ def validate_pattern(val): question["validate"] = validate_pattern - # WORKAROUND - Questionary cannot have a default position in a list - # For now, move the default option to the top. - if question["type"] == "list" and "default" in question: - try: - question["choices"].remove(question["default"]) - question["choices"].insert(0, question["default"]) - except ValueError: - log.warning( - "Default value `{}` not found in list of choices: {}".format( - question["default"], ", ".join(question["choices"]) - ) - ) - ### End of workaround code - return question def print_param_header(self, param_id, param_obj): @@ -691,3 +681,5 @@ def launch_workflow(self): if Confirm.ask("Do you want to run this command now? "): log.info("Launching workflow! :rocket:") subprocess.call(self.nextflow_cmd, shell=True) + + From 92eae6dde30e70b2e3f0cf80bf6f762e27af89cd Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Fri, 27 Nov 2020 10:51:51 +0100 Subject: [PATCH 18/50] uncommented 'validate' tests in test_launch --- tests/test_launch.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/test_launch.py b/tests/test_launch.py index 1756a9b7bb..70c2b0a29b 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -267,9 +267,9 @@ def test_ob_to_questionary_enum(self): assert result["type"] == "list" assert result["default"] == "copy" assert result["choices"] == ["symlink", "rellink"] - assert result["validate"]("symlink") is True - assert result["validate"]("") is True - assert result["validate"]("not_allowed") == "Must be one of: symlink, rellink" + # assert result["validate"]("symlink") is True + # assert result["validate"]("") is True + # assert result["validate"]("not_allowed") == "Must be one of: symlink, rellink" def test_ob_to_questionary_pattern(self): """ Check converting a python dict to a pyenquirer format - with pattern """ From e12ab1c483ed74ea6a2ce0d9d51afcbf9505aa6a Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Fri, 27 Nov 2020 12:46:50 +0100 Subject: [PATCH 19/50] Updated style; fixed unsafe_promt in test --- nf_core/launch.py | 34 +++++++++++++++++----------------- tests/test_launch.py | 8 ++++---- 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 116f60d595..9d0b184bf1 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -21,19 +21,21 @@ log = logging.getLogger(__name__) -# Custom style for questionary -nfcore_question_style = Style([ - ('qmark', 'fg:#673ab7 bold'), # token in front of the question - ('question', 'bold'), # question text - ('answer', 'fg:#f44336 bold'), # submitted answer text behind the question - ('pointer', 'fg:#eff53b bold'), # pointer used in select and checkbox prompts - ('highlighted', 'fg:#673ab7 bold'), # pointed-at choice in select and checkbox prompts - ('selected', 'fg:#cc5454'), # style for a selected item of a checkbox - ('separator', 'fg:#cc5454'), # separator in lists - ('instruction', ''), # user instructions for select, rawselect, checkbox - ('text', ''), # plain text - ('disabled', 'fg:#858585 italic') # disabled choices for select and checkbox prompts -]) +# Custom style for questionary +nfcore_question_style = Style( + [ + ("qmark", "fg:#673ab7 bold"), # token in front of the question + ("question", "bold"), # question text + ("answer", "fg:#f44336 bold"), # submitted answer text behind the question + ("pointer", "fg:#eff53b bold"), # pointer used in select and checkbox prompts + ("highlighted", "fg:#673ab7 bold"), # pointed-at choice in select and checkbox prompts + ("selected", "fg:#cc5454"), # style for a selected item of a checkbox + ("separator", "fg:#cc5454"), # separator in lists + ("instruction", ""), # user instructions for select, rawselect, checkbox + ("text", ""), # plain text + ("disabled", "fg:#858585 italic"), # disabled choices for select and checkbox prompts + ] +) class Launch(object): @@ -261,7 +263,7 @@ def prompt_web_gui(self): "name": "use_web_gui", "message": "Choose launch method", "choices": ["Web based", "Command line"], - "default": "Web based" + "default": "Web based", } answer = questionary.unsafe_prompt([question], style=nfcore_question_style) return answer["use_web_gui"] == "Web based" @@ -600,7 +602,7 @@ def filter_range(val): # return True # return "Must be one of: {}".format(", ".join(param_obj["enum"])) - #question["validate"] = validate_enum + # question["validate"] = validate_enum # Validate pattern from schema if "pattern" in param_obj: @@ -681,5 +683,3 @@ def launch_workflow(self): if Confirm.ask("Do you want to run this command now? "): log.info("Launching workflow! :rocket:") subprocess.call(self.nextflow_cmd, shell=True) - - diff --git a/tests/test_launch.py b/tests/test_launch.py index 70c2b0a29b..fd2609634e 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -102,12 +102,12 @@ def test_ob_to_questionary_string(self): result = self.launcher.single_param_to_questionary("input", sc_obj) assert result == {"type": "input", "name": "input", "message": "input", "default": "data/*{1,2}.fastq.gz"} - @mock.patch("questionary.prompt", side_effect=[{"use_web_gui": "Web based"}]) + @mock.patch("questionary.unsafe_prompt", side_effect=[{"use_web_gui": "Web based"}]) def test_prompt_web_gui_true(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == True - @mock.patch("questionary.prompt", side_effect=[{"use_web_gui": "Command line"}]) + @mock.patch("questionary.unsafe_prompt", side_effect=[{"use_web_gui": "Command line"}]) def test_prompt_web_gui_false(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == False @@ -261,7 +261,7 @@ def test_ob_to_questionary_range(self): assert result["filter"]("") == "" def test_ob_to_questionary_enum(self): - """ Check converting a python dict to a pyenquirer format - with enum """ + """ Check converting a python dict to a questionary format - with enum """ sc_obj = {"type": "string", "default": "copy", "enum": ["symlink", "rellink"]} result = self.launcher.single_param_to_questionary("publish_dir_mode", sc_obj) assert result["type"] == "list" @@ -272,7 +272,7 @@ def test_ob_to_questionary_enum(self): # assert result["validate"]("not_allowed") == "Must be one of: symlink, rellink" def test_ob_to_questionary_pattern(self): - """ Check converting a python dict to a pyenquirer format - with pattern """ + """ Check converting a python dict to a questionary format - with pattern """ sc_obj = {"type": "string", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$"} result = self.launcher.single_param_to_questionary("email", sc_obj) assert result["type"] == "input" From f4a36b1d0a811781f3ecafa6a59245b69baaef6c Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Fri, 27 Nov 2020 13:49:20 +0100 Subject: [PATCH 20/50] Update nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md Co-authored-by: Phil Ewels --- .../{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index dd33cb54dd..2f42b314e4 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -83,7 +83,8 @@ More details are as follow ### Default Values -Default values should go in `nextflow.config` under the `params` scope, and `nextflow_schema.json` (latter with `nf-core schema build .`) +Parameters should be initialised / defined with default values in `nextflow.config` under the `params` scope. +Once there, use `nf-core schema build .` to add to `nextflow_schema.json`. ### Default Processes Resource Requirements From 1aa1323e7ef3bc3d2d48f85759874d8169777e8e Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Fri, 27 Nov 2020 13:51:37 +0100 Subject: [PATCH 21/50] Update nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md Co-authored-by: Phil Ewels --- .../{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index 2f42b314e4..dba4f8592c 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -88,7 +88,7 @@ Once there, use `nf-core schema build .` to add to `nextflow_schema.json`. ### Default Processes Resource Requirements -Defining recommended 'minimum' resource requirements (CPUs/Memory) for a process should be defined in `conf/base.config`. This can be utilised within the process using `${task.cpu}` or `${task.memory}` variables in the `script:` block. +Defining sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. This can be utilised within the process using `${task.cpu}` or `${task.memory}` variables in the `script:` block. ### Naming Schemes From 8eaf4fc52c5f20d3b22dba536e283e43badfea29 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Fri, 27 Nov 2020 13:51:56 +0100 Subject: [PATCH 22/50] Update nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md Co-authored-by: Phil Ewels --- .../{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index dba4f8592c..7b2737ca55 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -99,7 +99,7 @@ Please use the following naming schemes, to make it easy to understand what is g ### Nextflow Version Bumping -If you have agreement from reviewers, you may bump the 'default' minimum version of nextflow (e.g. for testing), with `nf-core bump-version`. +If you have agreement from reviewers, you may bump the 'default' minimum version of nextflow (e.g. for testing), with: `nf-core bump-version --nextflow . [min-nf-version]` ### Software Version Reporting From 5a1b51a3cfeccca3771062b86d16a514acfe3fde Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Fri, 27 Nov 2020 13:54:41 +0100 Subject: [PATCH 23/50] Apply suggestions from code review - now I remembered about batch suggestions :facepalm: Co-authored-by: Phil Ewels --- .../.github/ISSUE_TEMPLATE/bug_report.md | 12 ++++-------- .../.github/PULL_REQUEST_TEMPLATE.md | 8 +++----- .../{{cookiecutter.name_noslash}}/README.md | 9 +++++---- 3 files changed, 12 insertions(+), 17 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md index 5f67bb76c7..64aa8d22b0 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md @@ -15,13 +15,10 @@ Please delete this text and anything that's not relevant from the template below ## Check Documentation -Have you checked in the following places for your error?: +I have checked the following places for your error: -- [ ] [nf-co.re troubleshooting(https://nf-co.re/usage/troubleshooting) -- [ ] [{{ cookiecutter.name }}-specific documentation](https://nf-co.re/{{ cookiecutter.name }}/usage) - -Please also check the the corresponding version's documentation, if not -testing the latest release. +- [ ] [nf-core website: troubleshooting](https://nf-co.re/usage/troubleshooting) +- [ ] [{{ cookiecutter.name }} pipeline documentation](https://nf-co.re/{{ cookiecutter.name }}/usage) ## Description of the bug @@ -43,8 +40,7 @@ Steps to reproduce the behaviour: Have you provided the following extra information/files: - [ ] The command used to run the pipeline -- [ ] The `.nextflow.log` file (which is a hidden file in whichever place you _ran_ - the pipeline from - not necessarily in the output directory!) +- [ ] The `.nextflow.log` file ## System diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md index eff042c5a2..8c4b16d0f7 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md @@ -15,14 +15,12 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/{{ cookiecut - [ ] This comment contains a description of changes (with reason). - [ ] If you've fixed a bug or added code that should be tested, add tests! - - [ ] If you've added a new tool - add to the software_versions process and a regex to `scrape_software_versions.py` - - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/{{ cookiecutter.name }}/tree/master/.github/CONTRIBUTING.md) - - [ ] If necessary, also make a PR on the {{ cookiecutter.name }} _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. + - [ ] If you've added a new tool - add to the software_versions process and a regex to `scrape_software_versions.py` + - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/{{ cookiecutter.name }}/tree/master/.github/CONTRIBUTING.md) + - [ ] If necessary, also make a PR on the {{ cookiecutter.name }} _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. - [ ] Make sure your code lints (`nf-core lint .`). - [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`). - [ ] Usage Documentation in `docs/usage.md` is updated. - [ ] Output Documentation in `docs/output.md` is updated. - [ ] `CHANGELOG.md` is updated. - [ ] `README.md` is updated (including new tool citations and authors/contributors). - -**Learn more about contributing:** https://github.com/nf-core/eager/tree/master/.github/CONTRIBUTING.md diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index 2f0a26df62..f50a928526 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -12,7 +12,8 @@ ## Introduction -**{{ cookiecutter.name }}** is a bioinformatics best-practise analysis pipeline for + +**{{ cookiecutter.name }}** is a bioinformatics best-practise analysis pipeline for The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It comes with docker containers making installation trivial and results highly reproducible. @@ -44,7 +45,7 @@ See [usage docs](https://nf-co.re/{{ cookiecutter.short_name }}/usage) for all o By default, the pipeline currently performs the following: - + * Sequencing quality control (`FastQC`) * Overall pipeline run summaries (`MultiQC`) @@ -62,7 +63,7 @@ The {{ cookiecutter.name }} pipeline comes with documentation about the pipeline We thank the following people for their extensive assistance in the development of this pipeline: - + ## Contributions and Support @@ -86,4 +87,4 @@ You can cite the `nf-core` publication as follows: In addition, references of tools and data used in this pipeline are as follows: - \ No newline at end of file + From be79294ecdb89733492c52ef62483cb648fe8460 Mon Sep 17 00:00:00 2001 From: James Fellows Yates Date: Fri, 27 Nov 2020 14:05:21 +0100 Subject: [PATCH 24/50] Final tweaks from Phil's comments --- .../.github/CONTRIBUTING.md | 33 +++++++++---------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index 7b2737ca55..329d2d5e48 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -31,14 +31,14 @@ Typically, pull-requests are only fully reviewed when these tests are passing, t There are typically two types of tests that run: -### Lint Tests +### Lint tests `nf-core` has a [set of guidelines](https://nf-co.re/developers/guidelines) which all pipelines must adhere to. To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint ` command. If any failures or warnings are encountered, please follow the listed URL for more documentation. -### Pipeline Tests +### Pipeline tests Each `nf-core` pipeline should be set up with a minimal set of test-data. `GitHub Actions` then runs the pipeline on this data to ensure that it exits successfully. @@ -53,17 +53,17 @@ These tests are run both with the latest available version of `Nextflow` and als * Fix the bug, and bump version (X.Y.Z+1). * A PR should be made on `master` from patch to directly this particular bug. -## Getting Help +## Getting help For further information/help, please consult the [{{ cookiecutter.name }} documentation](https://nf-co.re/{{ cookiecutter.short_name }}/usage) and don't hesitate to get in touch on the nf-core Slack [#{{ cookiecutter.short_name }}](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) channel ([join our Slack here](https://nf-co.re/join/slack)). -## Pipeline Contribution Conventions +## Pipeline contribution conventions To make the {{ cookiecutter.short_name }} code and processing logic more understandable for new contributors and to ensure quality, we semi-standardise the way the code and other contributions are written. -### Adding a New Module +### Adding a new step -If you wish to contribute a new module, please use the following coding standards: +If you wish to contribute a new step, please use the following coding standards: 1. Define the corresponding input channel into your new process from the expected previous process channel 2. Write the process block (see below). @@ -79,29 +79,30 @@ If you wish to contribute a new module, please use the following coding standard 12. Update MultiQC config `assets/multiqc_config.yaml` so relevant suffixes, name clean up, General Statistics Table column order, and module figures are in the right order. 13. Optional: Add any descriptions of MultiQC report sections and output files to `docs/output.md`. -More details are as follow +### Default values -### Default Values +Parameters should be initialised / defined with default values in `nextflow.config` under the `params` scope. -Parameters should be initialised / defined with default values in `nextflow.config` under the `params` scope. Once there, use `nf-core schema build .` to add to `nextflow_schema.json`. -### Default Processes Resource Requirements +### Default processes resource requirements -Defining sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. This can be utilised within the process using `${task.cpu}` or `${task.memory}` variables in the `script:` block. +Defining sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A popular example that is used in many nf-core pipelines is that of [nf-core/rnaseq](https://github.com/nf-core/rnaseq/). Process specific requirements (i.e. outside of generic labels), can be specified with the `withName:` selector (such as in [nf-core/eager](https://github.com/nf-core/eager/)). -### Naming Schemes +These can be utilised within the process itself using `${task.cpu}` or `${task.memory}` variables in the `script:` block. + +### Naming schemes Please use the following naming schemes, to make it easy to understand what is going where. * initial process channel: `ch_output_from_` * intermediate and terminal channels: `ch__for_` -### Nextflow Version Bumping +### Nextflow version bumping If you have agreement from reviewers, you may bump the 'default' minimum version of nextflow (e.g. for testing), with: `nf-core bump-version --nextflow . [min-nf-version]` -### Software Version Reporting +### Software version reporting If you add a new tool to the pipeline, please ensure you add the information of the tool to the `get_software_version` process. @@ -122,8 +123,6 @@ You then need to edit the script `bin/scrape_software_versions.py` to 1. add a (python) regex for your tools --version output (as in stored in the `v_.txt` file), to ensure the version is reported as a `v` and the version number e.g. `v2.1.1` 2. add a HTML block entry to the `OrderedDict` for formatting in MultiQC. -> If a tool does not unfortunately offer any printing of version data, you may add this 'manually' e.g. with `echo "v1.1" > v_.txt` - -### Images and Figures +### Images and figures For overview images and other documents we follow the nf-core [style guidelines and examples](https://nf-co.re/developers/design_guidelines). From a8c7eee97a5c615b2d3273d79600a9a13d17a713 Mon Sep 17 00:00:00 2001 From: James Fellows Yates Date: Fri, 27 Nov 2020 14:08:36 +0100 Subject: [PATCH 25/50] Linting --- .../pipeline-template/{{cookiecutter.name_noslash}}/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index f50a928526..8f3b5b3cb6 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -13,7 +13,7 @@ ## Introduction -**{{ cookiecutter.name }}** is a bioinformatics best-practise analysis pipeline for +**{{ cookiecutter.name }}** is a bioinformatics best-practise analysis pipeline for The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It comes with docker containers making installation trivial and results highly reproducible. From 836814e0b716b5057382f585f4574bb0d51be047 Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Fri, 27 Nov 2020 15:44:55 +0100 Subject: [PATCH 26/50] Deleted validation for 'enum' objects --- nf_core/launch.py | 15 +++------------ tests/test_launch.py | 3 --- 2 files changed, 3 insertions(+), 15 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 9d0b184bf1..c97f629a7a 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -2,6 +2,7 @@ """ Launch a pipeline, interactively collecting params """ from __future__ import print_function +from questionary import constants from rich.console import Console from rich.markdown import Markdown from rich.prompt import Confirm @@ -27,9 +28,9 @@ ("qmark", "fg:#673ab7 bold"), # token in front of the question ("question", "bold"), # question text ("answer", "fg:#f44336 bold"), # submitted answer text behind the question - ("pointer", "fg:#eff53b bold"), # pointer used in select and checkbox prompts + ("pointer", "fg:#f5d142 bold"), # pointer used in select and checkbox prompts ("highlighted", "fg:#673ab7 bold"), # pointed-at choice in select and checkbox prompts - ("selected", "fg:#cc5454"), # style for a selected item of a checkbox + ("selected", "fg:#f5d142"), # style for a selected item of a checkbox ("separator", "fg:#cc5454"), # separator in lists ("instruction", ""), # user instructions for select, rawselect, checkbox ("text", ""), # plain text @@ -37,7 +38,6 @@ ] ) - class Launch(object): """ Class to hold config option to launch a pipeline """ @@ -594,15 +594,6 @@ def filter_range(val): question["type"] = "list" question["choices"] = param_obj["enum"] - # # Validate enum from schema - # def validate_enum(val): - # if val == "": - # return True - # if val in param_obj["enum"]: - # return True - # return "Must be one of: {}".format(", ".join(param_obj["enum"])) - - # question["validate"] = validate_enum # Validate pattern from schema if "pattern" in param_obj: diff --git a/tests/test_launch.py b/tests/test_launch.py index fd2609634e..56c9a412b0 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -267,9 +267,6 @@ def test_ob_to_questionary_enum(self): assert result["type"] == "list" assert result["default"] == "copy" assert result["choices"] == ["symlink", "rellink"] - # assert result["validate"]("symlink") is True - # assert result["validate"]("") is True - # assert result["validate"]("not_allowed") == "Must be one of: symlink, rellink" def test_ob_to_questionary_pattern(self): """ Check converting a python dict to a questionary format - with pattern """ From 4bdab7c911946c3f116f4fed50a3af9ea68503a3 Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Fri, 27 Nov 2020 15:47:38 +0100 Subject: [PATCH 27/50] made black happy again --- nf_core/launch.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index c97f629a7a..95be8382f6 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -38,6 +38,7 @@ ] ) + class Launch(object): """ Class to hold config option to launch a pipeline """ @@ -594,7 +595,6 @@ def filter_range(val): question["type"] = "list" question["choices"] = param_obj["enum"] - # Validate pattern from schema if "pattern" in param_obj: From eba57a112eade8c2e0dcbe566079850dd073cb0c Mon Sep 17 00:00:00 2001 From: matthiasho Date: Sat, 28 Nov 2020 16:53:29 +0100 Subject: [PATCH 28/50] fix "range"-type parameters --- nf_core/launch.py | 9 --------- nf_core/schema.py | 4 ++-- tests/test_launch.py | 2 +- 3 files changed, 3 insertions(+), 12 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 8d64a612b3..2fc8045a63 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -568,7 +568,6 @@ def filter_integer(val): question["filter"] = filter_integer - if param_obj.get("type") == "range": # Validate range type def validate_range(val): try: @@ -585,14 +584,6 @@ def validate_range(val): question["validate"] = validate_range - # Filter returned value - def filter_range(val): - if val.strip() == "": - return "" - return float(val) - - question["filter"] = filter_range - if "enum" in param_obj: # Use a selection list instead of free text input question["type"] = "list" diff --git a/nf_core/schema.py b/nf_core/schema.py index 0df35e3ea3..1fea917283 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -424,7 +424,7 @@ def prompt_remove_schema_notfound_config(self, p_key): if self.no_prompts or self.schema_from_scratch: return True if Confirm.ask( - ":question: Unrecognised [white bold]'params.{}'[/] found in schema but not pipeline! [yellow]Remove it?".format( + ":question: Unrecognised [white bold]'params.{}'[/] found in the schema but not in the pipeline config! [yellow]Remove it?".format( p_key ) ): @@ -443,7 +443,7 @@ def add_schema_found_configs(self): self.no_prompts or self.schema_from_scratch or Confirm.ask( - ":sparkles: Found [white bold]'params.{}'[/] in pipeline but not in schema. [blue]Add to pipeline schema?".format( + ":sparkles: Found [white bold]'params.{}'[/] in the pipeline config, but not in the schema. [blue]Add to pipeline schema?".format( p_key ) ) diff --git a/tests/test_launch.py b/tests/test_launch.py index 70acc982a4..21c16e7a72 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -248,7 +248,7 @@ def test_ob_to_pyinquirer_integer(self): def test_ob_to_pyinquirer_range(self): """ Check converting a python dict to a pyenquirer format - with enum """ - sc_obj = {"type": "range", "minimum": "10", "maximum": "20", "default": 15} + sc_obj = {"type": "number", "minimum": "10", "maximum": "20", "default": 15} result = self.launcher.single_param_to_pyinquirer("broad_cutoff", sc_obj) assert result["type"] == "input" assert result["default"] == "15" From c47e86a4b95b0f79e5758ed5c3c05b038f505d5b Mon Sep 17 00:00:00 2001 From: matthiasho Date: Sat, 28 Nov 2020 17:26:33 +0100 Subject: [PATCH 29/50] fix for validate_range position --- nf_core/launch.py | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 2fc8045a63..1a43b460e0 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -538,6 +538,22 @@ def validate_number(val): question["validate"] = validate_number + # Validate range type + def validate_range(val): + try: + if val.strip() == "": + return True + fval = float(val) + if "minimum" in param_obj and fval < float(param_obj["minimum"]): + return "Must be greater than or equal to {}".format(param_obj["minimum"]) + if "maximum" in param_obj and fval > float(param_obj["maximum"]): + return "Must be less than or equal to {}".format(param_obj["maximum"]) + return True + except ValueError: + return "Must be a number" + + question["validate"] = validate_range + # Filter returned value def filter_number(val): if val.strip() == "": @@ -568,22 +584,6 @@ def filter_integer(val): question["filter"] = filter_integer - # Validate range type - def validate_range(val): - try: - if val.strip() == "": - return True - fval = float(val) - if "minimum" in param_obj and fval < float(param_obj["minimum"]): - return "Must be greater than or equal to {}".format(param_obj["minimum"]) - if "maximum" in param_obj and fval > float(param_obj["maximum"]): - return "Must be less than or equal to {}".format(param_obj["maximum"]) - return True - except ValueError: - return "Must be a number" - - question["validate"] = validate_range - if "enum" in param_obj: # Use a selection list instead of free text input question["type"] = "list" From d3c1ac9112d2b1469db12e1d498bbc7c8a5e1b7b Mon Sep 17 00:00:00 2001 From: matthiasho Date: Sat, 28 Nov 2020 17:27:03 +0100 Subject: [PATCH 30/50] update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ebd55dbe8..9bdfb2b408 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ### Tools helper code +* json-schema: make paramters of type `range` to `number`.[[#738](https://github.com/nf-core/tools/issues/738)] * Respect `$NXF_HOME` when looking for pipelines with `nf-core list` [[#798](https://github.com/nf-core/tools/issues/798)] ### Template From 01be190184f99e7df783f72e4ee23bb8e80f2f4a Mon Sep 17 00:00:00 2001 From: Kevin Menden Date: Sun, 29 Nov 2020 08:48:01 +0100 Subject: [PATCH 31/50] Update .gitignore Co-authored-by: Phil Ewels --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 1d244caa14..366282b683 100644 --- a/.gitignore +++ b/.gitignore @@ -116,4 +116,4 @@ pip-wheel-metadata/nf_core.dist-info/top_level.txt pip-wheel-metadata/nf_core.dist-info/METADATA pip-wheel-metadata/nf_core.dist-info/LICENSE pip-wheel-metadata/nf_core.dist-info/entry_points.txt -.vscode/settings.json +.vscode From 424683c5579743e44996879e29d0e38dfb6b89ac Mon Sep 17 00:00:00 2001 From: Kevin Menden Date: Sun, 29 Nov 2020 08:48:09 +0100 Subject: [PATCH 32/50] Update .gitignore Co-authored-by: Phil Ewels --- .gitignore | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/.gitignore b/.gitignore index 366282b683..77ce81a93b 100644 --- a/.gitignore +++ b/.gitignore @@ -112,8 +112,5 @@ ENV/ # Jetbrains IDEs .idea -pip-wheel-metadata/nf_core.dist-info/top_level.txt -pip-wheel-metadata/nf_core.dist-info/METADATA -pip-wheel-metadata/nf_core.dist-info/LICENSE -pip-wheel-metadata/nf_core.dist-info/entry_points.txt +pip-wheel-metadata .vscode From d2b23d66a23e19936c7a76fadbb160ded85a342b Mon Sep 17 00:00:00 2001 From: matthiasho Date: Mon, 30 Nov 2020 11:55:04 +0100 Subject: [PATCH 33/50] move max-min validation into `validate_number` --- nf_core/launch.py | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 1a43b460e0..82e7d3898b 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -531,28 +531,16 @@ def validate_number(val): if val.strip() == "": return True float(val) - except ValueError: - return "Must be a number" - else: - return True - - question["validate"] = validate_number - - # Validate range type - def validate_range(val): - try: - if val.strip() == "": - return True - fval = float(val) if "minimum" in param_obj and fval < float(param_obj["minimum"]): return "Must be greater than or equal to {}".format(param_obj["minimum"]) if "maximum" in param_obj and fval > float(param_obj["maximum"]): return "Must be less than or equal to {}".format(param_obj["maximum"]) - return True except ValueError: return "Must be a number" + else: + return True - question["validate"] = validate_range + question["validate"] = validate_number # Filter returned value def filter_number(val): From c3cf0459c7982e954b31c078f0ce0a7cb0db05f4 Mon Sep 17 00:00:00 2001 From: matthiasho Date: Mon, 30 Nov 2020 12:04:24 +0100 Subject: [PATCH 34/50] fix missing variable declaration --- nf_core/launch.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 82e7d3898b..e358781dd5 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -530,7 +530,7 @@ def validate_number(val): try: if val.strip() == "": return True - float(val) + fval = float(val) if "minimum" in param_obj and fval < float(param_obj["minimum"]): return "Must be greater than or equal to {}".format(param_obj["minimum"]) if "maximum" in param_obj and fval > float(param_obj["maximum"]): From 45b28fb8d8131dc846eba25dc7c932de8dc55647 Mon Sep 17 00:00:00 2001 From: Kevin Menden Date: Tue, 1 Dec 2020 08:38:37 +0100 Subject: [PATCH 35/50] remove questionary.constants import Co-authored-by: Phil Ewels --- nf_core/launch.py | 1 - 1 file changed, 1 deletion(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 95be8382f6..4faba0cca6 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -2,7 +2,6 @@ """ Launch a pipeline, interactively collecting params """ from __future__ import print_function -from questionary import constants from rich.console import Console from rich.markdown import Markdown from rich.prompt import Confirm From 3099aa77f15242fd685fb252081d06a502ade5af Mon Sep 17 00:00:00 2001 From: Kevin Menden Date: Tue, 1 Dec 2020 08:42:08 +0100 Subject: [PATCH 36/50] Update CHANGELOG.md Co-authored-by: Phil Ewels --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7afac0ec0f..dd2b19902a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## v1.13dev -* swapped PyInquirer with questionary for command line questions in launch.py +* Swapped PyInquirer with questionary for command line questions in `launch.py` ### Tools helper code From 2a35e5834f22e6ea6a7e12dd49b506d7f9219ec2 Mon Sep 17 00:00:00 2001 From: kevinmenden Date: Tue, 1 Dec 2020 08:51:35 +0100 Subject: [PATCH 37/50] Specified questionary version, added prompt_toolkit --- CHANGELOG.md | 3 ++- setup.py | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dd2b19902a..da54749fb8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,8 @@ ## v1.13dev -* Swapped PyInquirer with questionary for command line questions in `launch.py` +* Swapped PyInquirer with questionary for command line questions in `launch.py` [[#726 +]](https://github.com/nf-core/tools/issues/726) ### Tools helper code diff --git a/setup.py b/setup.py index 4f727720e7..5d8cd4aa46 100644 --- a/setup.py +++ b/setup.py @@ -36,7 +36,8 @@ "GitPython", "jinja2", "jsonschema", - "questionary", + "questionary>=1.8.0", + "prompt_toolkit", "pyyaml", "requests", "requests_cache", From 622a5575c4c8dafda73ada32175e86bcd8b92359 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 2 Dec 2020 10:51:12 +0100 Subject: [PATCH 38/50] Use ANSI colours, show value in select list * Use ANSI colours instead of hex values, so that colours honour the terminal settings (and match the nf-core logo) * Show the value for each parameter in the select list - either the default from the schema or the entered value --- nf_core/launch.py | 55 ++++++++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 25 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 4faba0cca6..b0a3d99156 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -10,30 +10,30 @@ import json import logging import os +import prompt_toolkit import questionary import re import subprocess import textwrap import webbrowser -from prompt_toolkit.styles import Style import nf_core.schema, nf_core.utils log = logging.getLogger(__name__) # Custom style for questionary -nfcore_question_style = Style( +nfcore_question_style = prompt_toolkit.styles.Style( [ - ("qmark", "fg:#673ab7 bold"), # token in front of the question + ("qmark", "fg:ansiblue bold"), # token in front of the question ("question", "bold"), # question text - ("answer", "fg:#f44336 bold"), # submitted answer text behind the question - ("pointer", "fg:#f5d142 bold"), # pointer used in select and checkbox prompts - ("highlighted", "fg:#673ab7 bold"), # pointed-at choice in select and checkbox prompts - ("selected", "fg:#f5d142"), # style for a selected item of a checkbox - ("separator", "fg:#cc5454"), # separator in lists + ("answer", "fg:ansigreen nobold"), # submitted answer text behind the question + ("pointer", "fg:ansiyellow bold"), # pointer used in select and checkbox prompts + ("highlighted", "fg:ansiblue bold"), # pointed-at choice in select and checkbox prompts + ("selected", "fg:ansigreen noreverse"), # style for a selected item of a checkbox + ("separator", "fg:ansiblack"), # separator in lists ("instruction", ""), # user instructions for select, rawselect, checkbox ("text", ""), # plain text - ("disabled", "fg:#858585 italic"), # disabled choices for select and checkbox prompts + ("disabled", "fg:gray italic"), # disabled choices for select and checkbox prompts ] ) @@ -425,24 +425,29 @@ def prompt_group(self, group_id, group_obj): Returns: Dict of param_id:val answers """ - question = { - "type": "list", - "name": group_id, - "message": group_obj.get("title", group_id), - "choices": ["Continue >>", questionary.Separator()], - } - - for param_id, param in group_obj["properties"].items(): - if not param.get("hidden", False) or self.show_hidden: - question["choices"].append(param_id) - - # Skip if all questions hidden - if len(question["choices"]) == 2: - return {} - while_break = False answers = {} while not while_break: + question = { + "type": "list", + "name": group_id, + "message": group_obj.get("title", group_id), + "choices": ["Continue >>", questionary.Separator()], + } + + for param_id, param in group_obj["properties"].items(): + if not param.get("hidden", False) or self.show_hidden: + q_title = param_id + if param_id in answers: + q_title += " [{}]".format(answers[param_id]) + elif "default" in param: + q_title += " [{}]".format(param["default"]) + question["choices"].append(questionary.Choice(title=q_title, value=param_id)) + + # Skip if all questions hidden + if len(question["choices"]) == 2: + return {} + self.print_param_header(group_id, group_obj) answer = questionary.unsafe_prompt([question], style=nfcore_question_style) if answer[group_id] == "Continue >>": @@ -452,7 +457,7 @@ def prompt_group(self, group_id, group_obj): req_default = self.schema_obj.input_params.get(p_required, "") req_answer = answers.get(p_required, "") if req_default == "" and req_answer == "": - log.error("'{}' is required.".format(p_required)) + log.error("'--{}' is required.".format(p_required)) while_break = False else: param_id = answer[group_id] From b0b407f3ffc8b4045af68a749b0ce6a9e4cd280f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 2 Dec 2020 10:53:46 +0100 Subject: [PATCH 39/50] Fix changelog link --- CHANGELOG.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index da54749fb8..00e56bfe95 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,8 +2,9 @@ ## v1.13dev -* Swapped PyInquirer with questionary for command line questions in `launch.py` [[#726 -]](https://github.com/nf-core/tools/issues/726) +* Swapped PyInquirer with questionary for command line questions in `launch.py` [[#726](https://github.com/nf-core/tools/issues/726)] + * This should fix conda installation issues that some people had been hitting + * The change also allows other improvements to the UI ### Tools helper code From 3ab1051e1604790dcfb509182b71ecfff7f64008 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 2 Dec 2020 12:36:20 +0100 Subject: [PATCH 40/50] Update CHANGELOG.md --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9bdfb2b408..7448b7cb4b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ ### Tools helper code -* json-schema: make paramters of type `range` to `number`.[[#738](https://github.com/nf-core/tools/issues/738)] +* Pipeline schema: make parameters of type `range` to `number`. [[#738](https://github.com/nf-core/tools/issues/738)] * Respect `$NXF_HOME` when looking for pipelines with `nf-core list` [[#798](https://github.com/nf-core/tools/issues/798)] ### Template From 598cf1da52d1a67a5ea74d480241b03422336b94 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Wed, 2 Dec 2020 14:29:07 +0100 Subject: [PATCH 41/50] Update CONTRIBUTING.md --- .../{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index 329d2d5e48..d866eccd11 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -87,7 +87,8 @@ Once there, use `nf-core schema build .` to add to `nextflow_schema.json`. ### Default processes resource requirements -Defining sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A popular example that is used in many nf-core pipelines is that of [nf-core/rnaseq](https://github.com/nf-core/rnaseq/). Process specific requirements (i.e. outside of generic labels), can be specified with the `withName:` selector (such as in [nf-core/eager](https://github.com/nf-core/eager/)). +Defining sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in [nf-core/rnaseq](https://github.com/nf-core/rnaseq/), where the base config has the process default as a single core-process, and then different levels of multi-core jobs for increasingly large memory requirements defined with standard labels. +Process specific requirements (i.e. outside of generic labels), can be specified with the `withName:` selector (such as in [nf-core/eager](https://github.com/nf-core/eager/)). These can be utilised within the process itself using `${task.cpu}` or `${task.memory}` variables in the `script:` block. From f569bac1b62d69504434ef5ed6ab5de595c8e547 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 3 Dec 2020 00:28:16 +0100 Subject: [PATCH 42/50] Add minimum version for prompt_toolkit --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 5d8cd4aa46..aaf543a065 100644 --- a/setup.py +++ b/setup.py @@ -37,7 +37,7 @@ "jinja2", "jsonschema", "questionary>=1.8.0", - "prompt_toolkit", + "prompt_toolkit>=3.0.3", "pyyaml", "requests", "requests_cache", From 63dbb9fe8802dc0a7207cbf8a4180c76c80391bf Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Thu, 3 Dec 2020 08:40:39 +0100 Subject: [PATCH 43/50] Update nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md Co-authored-by: mashehu --- .../{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index d866eccd11..55c9c8012d 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -87,7 +87,7 @@ Once there, use `nf-core schema build .` to add to `nextflow_schema.json`. ### Default processes resource requirements -Defining sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in [nf-core/rnaseq](https://github.com/nf-core/rnaseq/), where the base config has the process default as a single core-process, and then different levels of multi-core jobs for increasingly large memory requirements defined with standard labels. +Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in [nf-core/rnaseq](https://github.com/nf-core/rnaseq/), where the base config has the process default as a single core-process, and then different levels of multi-core jobs for increasingly large memory requirements defined with standard labels. Process specific requirements (i.e. outside of generic labels), can be specified with the `withName:` selector (such as in [nf-core/eager](https://github.com/nf-core/eager/)). These can be utilised within the process itself using `${task.cpu}` or `${task.memory}` variables in the `script:` block. From 08945acbb7ea9ea924be29c5955e569d720bcc81 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Thu, 3 Dec 2020 08:42:38 +0100 Subject: [PATCH 44/50] Update CONTRIBUTING.md --- .../{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index 55c9c8012d..ac6ea36e90 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -87,8 +87,7 @@ Once there, use `nf-core schema build .` to add to `nextflow_schema.json`. ### Default processes resource requirements -Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in [nf-core/rnaseq](https://github.com/nf-core/rnaseq/), where the base config has the process default as a single core-process, and then different levels of multi-core jobs for increasingly large memory requirements defined with standard labels. -Process specific requirements (i.e. outside of generic labels), can be specified with the `withName:` selector (such as in [nf-core/eager](https://github.com/nf-core/eager/)). +Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in [nf-core/rnaseq](https://github.com/nf-core/rnaseq/blob/master/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels. These can be utilised within the process itself using `${task.cpu}` or `${task.memory}` variables in the `script:` block. From cfae735856972c37514407c46442a99ee6c042b2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 3 Dec 2020 11:06:10 +0100 Subject: [PATCH 45/50] Catch FileNotFoundError when looking for TODO comments Fixes nf-core/tools#796 Also switched the log message from error to critical when halting linting due to failed tests. --- nf_core/lint.py | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 74c5b170a4..e057bc38b6 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -239,7 +239,7 @@ def lint_pipeline(self, release_mode=False): log.debug("Running lint test: {}".format(fun_name)) getattr(self, fun_name)() if len(self.failed) > 0: - log.error("Found test failures in `{}`, halting lint run.".format(fun_name)) + log.critical("Found test failures in `{}`, halting lint run.".format(fun_name)) break def check_files_exist(self): @@ -1241,17 +1241,25 @@ def check_cookiecutter_strings(self): num_files = 0 for fn in list_of_files: num_files += 1 - with io.open(fn, "r", encoding="latin1") as fh: - lnum = 0 - for l in fh: - lnum += 1 - cc_matches = re.findall(r"{{\s*cookiecutter[^}]*}}", l) - if len(cc_matches) > 0: - for cc_match in cc_matches: - self.failed.append( - (13, "Found a cookiecutter template string in `{}` L{}: {}".format(fn, lnum, cc_match)) - ) - num_matches += 1 + try: + with io.open(fn, "r", encoding="latin1") as fh: + lnum = 0 + for l in fh: + lnum += 1 + cc_matches = re.findall(r"{{\s*cookiecutter[^}]*}}", l) + if len(cc_matches) > 0: + for cc_match in cc_matches: + self.failed.append( + ( + 13, + "Found a cookiecutter template string in `{}` L{}: {}".format( + fn, lnum, cc_match + ), + ) + ) + num_matches += 1 + except FileNotFoundError as e: + log.warn("`git ls-files` returned '{}' but could not open it!".format(fn)) if num_matches == 0: self.passed.append((13, "Did not find any cookiecutter template strings ({} files)".format(num_files))) From 80e6fddaf271afcb8b86c4d62dd529031ed1f1d2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 3 Dec 2020 11:27:11 +0100 Subject: [PATCH 46/50] Changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4726acaad2..f583d4f5be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,8 @@ ### Linting +* Fix linting crash when a file deleted but not yet staged in git [[#796](https://github.com/nf-core/tools/issues/796)] + ### Other ## [v1.12 - Mercury Weasel](https://github.com/nf-core/tools/releases/tag/1.12) - [2020-11-19] From 7e91bfc579c1153a3fdc6ab003301d80496169c1 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 3 Dec 2020 16:05:57 +0100 Subject: [PATCH 47/50] Bump version for v.1.12.1 release --- CHANGELOG.md | 19 ++++++++----------- setup.py | 2 +- 2 files changed, 9 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f583d4f5be..b61e50839f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,24 +1,21 @@ # nf-core/tools: Changelog -## v1.13dev +## [v1.12.1 - Silver Dolphin](https://github.com/nf-core/tools/releases/tag/1.12.1) - [2020-12-03] -* Swapped PyInquirer with questionary for command line questions in `launch.py` [[#726](https://github.com/nf-core/tools/issues/726)] - * This should fix conda installation issues that some people had been hitting - * The change also allows other improvements to the UI +### Template + +* Finished switch from `$baseDir` to `$projectDir` in `iGenomes.conf` and `main.nf` + * Main fix is for `smail_fields` which was a bug introduced in the previous release. Sorry about that! ### Tools helper code * Pipeline schema: make parameters of type `range` to `number`. [[#738](https://github.com/nf-core/tools/issues/738)] * Respect `$NXF_HOME` when looking for pipelines with `nf-core list` [[#798](https://github.com/nf-core/tools/issues/798)] - -### Template - -### Linting - +* Swapped PyInquirer with questionary for command line questions in `launch.py` [[#726](https://github.com/nf-core/tools/issues/726)] + * This should fix conda installation issues that some people had been hitting + * The change also allows other improvements to the UI * Fix linting crash when a file deleted but not yet staged in git [[#796](https://github.com/nf-core/tools/issues/796)] -### Other - ## [v1.12 - Mercury Weasel](https://github.com/nf-core/tools/releases/tag/1.12) - [2020-11-19] ### Tools helper code diff --git a/setup.py b/setup.py index aaf543a065..7cd0ebbe25 100644 --- a/setup.py +++ b/setup.py @@ -3,7 +3,7 @@ from setuptools import setup, find_packages import sys -version = "1.13dev" +version = "1.12.1" with open("README.md") as f: readme = f.read() From f226eb49427d3e5533da96bdf110decb45de2d8d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 3 Dec 2020 16:07:43 +0100 Subject: [PATCH 48/50] Apply suggestions from code review --- .../.github/CONTRIBUTING.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index ac6ea36e90..8bedc3996e 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -59,7 +59,7 @@ For further information/help, please consult the [{{ cookiecutter.name }} docume ## Pipeline contribution conventions -To make the {{ cookiecutter.short_name }} code and processing logic more understandable for new contributors and to ensure quality, we semi-standardise the way the code and other contributions are written. +To make the {{ cookiecutter.name }} code and processing logic more understandable for new contributors and to ensure quality, we semi-standardise the way the code and other contributions are written. ### Adding a new step @@ -87,9 +87,9 @@ Once there, use `nf-core schema build .` to add to `nextflow_schema.json`. ### Default processes resource requirements -Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in [nf-core/rnaseq](https://github.com/nf-core/rnaseq/blob/master/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels. +Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/%7B%7Bcookiecutter.name_noslash%7D%7D/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels. -These can be utilised within the process itself using `${task.cpu}` or `${task.memory}` variables in the `script:` block. +The process resources can be passed on to the tool dynamically within the process with the `${task.cpu}` and `${task.memory}` variables in the `script:` block. ### Naming schemes @@ -100,7 +100,7 @@ Please use the following naming schemes, to make it easy to understand what is g ### Nextflow version bumping -If you have agreement from reviewers, you may bump the 'default' minimum version of nextflow (e.g. for testing), with: `nf-core bump-version --nextflow . [min-nf-version]` +If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core bump-version --nextflow . [min-nf-version]` ### Software version reporting @@ -118,10 +118,10 @@ or --help | head -n 1 &> v_.txt 2>&1 || true ``` -You then need to edit the script `bin/scrape_software_versions.py` to +You then need to edit the script `bin/scrape_software_versions.py` to: -1. add a (python) regex for your tools --version output (as in stored in the `v_.txt` file), to ensure the version is reported as a `v` and the version number e.g. `v2.1.1` -2. add a HTML block entry to the `OrderedDict` for formatting in MultiQC. +1. Add a Python regex for your tool's `--version` output (as in stored in the `v_.txt` file), to ensure the version is reported as a `v` and the version number e.g. `v2.1.1` +2. Add a HTML entry to the `OrderedDict` for formatting in MultiQC. ### Images and figures From 4c4bc5cf0077d6294259b2d0a35a855ea783f8ac Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 3 Dec 2020 16:09:24 +0100 Subject: [PATCH 49/50] Apply markdownlint config settings to main tools config as well --- .github/markdownlint.yml | 7 +++++++ .../{{cookiecutter.name_noslash}}/.github/markdownlint.yml | 4 ++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/.github/markdownlint.yml b/.github/markdownlint.yml index a0eac5a096..793fead434 100644 --- a/.github/markdownlint.yml +++ b/.github/markdownlint.yml @@ -3,6 +3,13 @@ default: true line-length: false no-duplicate-header: siblings_only: true +no-inline-html: + allowed_elements: + - img + - p + - kbd + - details + - summary # tools only - the {{ jinja variables }} break URLs and cause this to error no-bare-urls: false # tools only - suppresses error messages for usage of $ in main README diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/markdownlint.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/markdownlint.yml index 0967bbbb81..8d7eb53b07 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/markdownlint.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/markdownlint.yml @@ -1,9 +1,9 @@ # Markdownlint configuration file -default: true, +default: true line-length: false no-duplicate-header: siblings_only: true -no-inline-html: +no-inline-html: allowed_elements: - img - p From a57a825e1411d65e72b9b11ae31a9d256c553707 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 3 Dec 2020 16:19:02 +0100 Subject: [PATCH 50/50] Changelog for nf-core/tools#788 --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b61e50839f..db034a4e99 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ * Finished switch from `$baseDir` to `$projectDir` in `iGenomes.conf` and `main.nf` * Main fix is for `smail_fields` which was a bug introduced in the previous release. Sorry about that! +* Ported a number of small content tweaks from nf-core/eager to the template [[#786](https://github.com/nf-core/tools/issues/786)] + * Better contributing documentation, more placeholders in documentation files, more relaxed markdownlint exceptions for certain HTML tags, more content for the PR and issue templates. ### Tools helper code