Adding and simplifying MaskPrimers options

[ggabernet]

• also store unproductive seqs
• make unproductive optional
• fix cregion primer annotation
• Merge branch 'dev' of https://github.com/ggabernet/nf-core-airrflow into dev
• simplify maskprimers score
• do not save presto intermediate fastq
• add maskprimers extract option

PR checklist

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
• If you've added a new tool - have you followed the pipeline conventions in the contribution docs
• If necessary, also make a PR on the nf-core/airrflow branch on the nf-core/test-datasets repository.
• Make sure your code lints (nf-core lint).
• Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
• Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
• Usage Documentation in docs/usage.md is updated.
• Output Documentation in docs/output.md is updated.
• CHANGELOG.md is updated.
• README.md is updated (including new tool citations and authors/contributors).

[github-actions]

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 9f26a8a

+| ✅ 299 tests passed       |+
#| ❔   7 tests were ignored |#
!| ❗   1 tests had warnings |!

❗ Test warnings:

• nextflow_config - Config manifest.version should end in dev: 4.1.0

❔ Tests ignored:

• files_exist - File is ignored: conf/igenomes.config
• nextflow_config - Config default ignored: params.miairr
• nextflow_config - Config default ignored: params.report_rmd
• nextflow_config - Config default ignored: params.report_css
• nextflow_config - Config default ignored: params.report_logo
• nextflow_config - Config default ignored: params.report_logo_img
• nextflow_config - Config default ignored: params.config_profile_url

✅ Tests passed:

• files_exist - File found: .gitattributes
• files_exist - File found: .gitignore
• files_exist - File found: .nf-core.yml
• files_exist - File found: .editorconfig
• files_exist - File found: .prettierignore
• files_exist - File found: .prettierrc.yml
• files_exist - File found: CHANGELOG.md
• files_exist - File found: CITATIONS.md
• files_exist - File found: CODE_OF_CONDUCT.md
• files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_exist - File found: nextflow_schema.json
• files_exist - File found: nextflow.config
• files_exist - File found: README.md
• files_exist - File found: .github/.dockstore.yml
• files_exist - File found: .github/CONTRIBUTING.md
• files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
• files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
• files_exist - File found: .github/workflows/branch.yml
• files_exist - File found: .github/workflows/ci.yml
• files_exist - File found: .github/workflows/linting_comment.yml
• files_exist - File found: .github/workflows/linting.yml
• files_exist - File found: assets/email_template.html
• files_exist - File found: assets/email_template.txt
• files_exist - File found: assets/sendmail_template.txt
• files_exist - File found: assets/nf-core-airrflow_logo_light.png
• files_exist - File found: conf/modules.config
• files_exist - File found: conf/test.config
• files_exist - File found: conf/test_full.config
• files_exist - File found: docs/images/nf-core-airrflow_logo_light.png
• files_exist - File found: docs/images/nf-core-airrflow_logo_dark.png
• files_exist - File found: docs/output.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/usage.md
• files_exist - File found: main.nf
• files_exist - File found: assets/multiqc_config.yml
• files_exist - File found: conf/base.config
• files_exist - File found: conf/igenomes_ignored.config
• files_exist - File found: .github/workflows/awstest.yml
• files_exist - File found: .github/workflows/awsfulltest.yml
• files_exist - File found: modules.json
• files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
• files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
• files_exist - File not found check: .github/workflows/push_dockerhub.yml
• files_exist - File not found check: .markdownlint.yml
• files_exist - File not found check: .nf-core.yaml
• files_exist - File not found check: .yamllint.yml
• files_exist - File not found check: bin/markdown_to_html.r
• files_exist - File not found check: conf/aws.config
• files_exist - File not found check: docs/images/nf-core-airrflow_logo.png
• files_exist - File not found check: lib/Checks.groovy
• files_exist - File not found check: lib/Completion.groovy
• files_exist - File not found check: lib/NfcoreTemplate.groovy
• files_exist - File not found check: lib/Utils.groovy
• files_exist - File not found check: lib/Workflow.groovy
• files_exist - File not found check: lib/WorkflowMain.groovy
• files_exist - File not found check: lib/WorkflowAirrflow.groovy
• files_exist - File not found check: parameters.settings.json
• files_exist - File not found check: pipeline_template.yml
• files_exist - File not found check: Singularity
• files_exist - File not found check: lib/nfcore_external_java_deps.jar
• files_exist - File not found check: .travis.yml
• nextflow_config - Found nf-schema plugin
• nextflow_config - Config variable found: manifest.name
• nextflow_config - Config variable found: manifest.nextflowVersion
• nextflow_config - Config variable found: manifest.description
• nextflow_config - Config variable found: manifest.version
• nextflow_config - Config variable found: manifest.homePage
• nextflow_config - Config variable found: timeline.enabled
• nextflow_config - Config variable found: trace.enabled
• nextflow_config - Config variable found: report.enabled
• nextflow_config - Config variable found: dag.enabled
• nextflow_config - Config variable found: process.cpus
• nextflow_config - Config variable found: process.memory
• nextflow_config - Config variable found: process.time
• nextflow_config - Config variable found: params.outdir
• nextflow_config - Config variable found: params.input
• nextflow_config - Config variable found: validation.help.enabled
• nextflow_config - Config variable found: manifest.mainScript
• nextflow_config - Config variable found: timeline.file
• nextflow_config - Config variable found: trace.file
• nextflow_config - Config variable found: report.file
• nextflow_config - Config variable found: dag.file
• nextflow_config - Config variable found: validation.help.beforeText
• nextflow_config - Config variable found: validation.help.afterText
• nextflow_config - Config variable found: validation.help.command
• nextflow_config - Config variable found: validation.summary.beforeText
• nextflow_config - Config variable found: validation.summary.afterText
• nextflow_config - Config variable (correctly) not found: params.nf_required_version
• nextflow_config - Config variable (correctly) not found: params.container
• nextflow_config - Config variable (correctly) not found: params.singleEnd
• nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
• nextflow_config - Config variable (correctly) not found: params.name
• nextflow_config - Config variable (correctly) not found: params.enable_conda
• nextflow_config - Config variable (correctly) not found: params.max_cpus
• nextflow_config - Config variable (correctly) not found: params.max_memory
• nextflow_config - Config variable (correctly) not found: params.max_time
• nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
• nextflow_config - Config variable (correctly) not found: params.validationLenientMode
• nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
• nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
• nextflow_config - Config timeline.enabled had correct value: true
• nextflow_config - Config report.enabled had correct value: true
• nextflow_config - Config trace.enabled had correct value: true
• nextflow_config - Config dag.enabled had correct value: true
• nextflow_config - Config manifest.name began with nf-core/
• nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
• nextflow_config - Config dag.file ended with .html
• nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
• nextflow_config - Config params.custom_config_version is set to master
• nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
• nextflow_config - Lines for loading custom profiles found
• nextflow_config - nextflow.config contains configuration profile test
• nextflow_config - Config default value correct: params.mode= fastq
• nextflow_config - Config default value correct: params.vprimer_start= 0
• nextflow_config - Config default value correct: params.cprimer_start= 0
• nextflow_config - Config default value correct: params.cprimer_position= R1
• nextflow_config - Config default value correct: params.umi_position= R1
• nextflow_config - Config default value correct: params.umi_length= -1
• nextflow_config - Config default value correct: params.umi_start= 0
• nextflow_config - Config default value correct: params.trim_fastq= true
• nextflow_config - Config default value correct: params.clip_r1= 0
• nextflow_config - Config default value correct: params.clip_r2= 0
• nextflow_config - Config default value correct: params.three_prime_clip_r1= 0
• nextflow_config - Config default value correct: params.three_prime_clip_r2= 0
• nextflow_config - Config default value correct: params.filterseq_q= 20
• nextflow_config - Config default value correct: params.primer_consensus= 0.6
• nextflow_config - Config default value correct: params.buildconsensus_maxerror= 0.1
• nextflow_config - Config default value correct: params.buildconsensus_maxgap= 0.5
• nextflow_config - Config default value correct: params.cluster_sets= true
• nextflow_config - Config default value correct: params.primer_r1_maxerror= 0.2
• nextflow_config - Config default value correct: params.primer_r2_maxerror= 0.2
• nextflow_config - Config default value correct: params.primer_maxlen= 50
• nextflow_config - Config default value correct: params.primer_r1_mask_mode= cut
• nextflow_config - Config default value correct: params.primer_r2_mask_mode= cut
• nextflow_config - Config default value correct: params.primer_r1_extract_len= 0
• nextflow_config - Config default value correct: params.primer_r2_extract_len= 0
• nextflow_config - Config default value correct: params.cregion_maxlen= 100
• nextflow_config - Config default value correct: params.cregion_maxerror= 0.3
• nextflow_config - Config default value correct: params.cregion_mask_mode= tag
• nextflow_config - Config default value correct: params.reassign= true
• nextflow_config - Config default value correct: params.productive_only= true
• nextflow_config - Config default value correct: params.save_databases= true
• nextflow_config - Config default value correct: params.reference_fasta= https://raw.githubusercontent.com/nf-core/test-datasets/airrflow/database-cache/imgtdb_base.zip
• nextflow_config - Config default value correct: params.reference_igblast= https://raw.githubusercontent.com/nf-core/test-datasets/airrflow/database-cache/igblast_base.zip
• nextflow_config - Config default value correct: params.isotype_column= c_call
• nextflow_config - Config default value correct: params.collapseby= sample_id
• nextflow_config - Config default value correct: params.clonal_threshold= auto
• nextflow_config - Config default value correct: params.cloneby= subject_id
• nextflow_config - Config default value correct: params.crossby= subject_id
• nextflow_config - Config default value correct: params.lineage_tree_builder= raxml
• nextflow_config - Config default value correct: params.lineage_tree_exec= /usr/local/bin/raxml-ng
• nextflow_config - Config default value correct: params.singlecell= single_cell
• nextflow_config - Config default value correct: params.igenomes_ignore= true
• nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
• nextflow_config - Config default value correct: params.custom_config_version= master
• nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
• nextflow_config - Config default value correct: params.publish_dir_mode= copy
• nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
• nextflow_config - Config default value correct: params.validate_params= true
• nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/airrflow/
• files_unchanged - .gitattributes matches the template
• files_unchanged - .prettierrc.yml matches the template
• files_unchanged - CODE_OF_CONDUCT.md matches the template
• files_unchanged - LICENSE matches the template
• files_unchanged - .github/.dockstore.yml matches the template
• files_unchanged - .github/CONTRIBUTING.md matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
• files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
• files_unchanged - .github/workflows/branch.yml matches the template
• files_unchanged - .github/workflows/linting_comment.yml matches the template
• files_unchanged - .github/workflows/linting.yml matches the template
• files_unchanged - assets/email_template.html matches the template
• files_unchanged - assets/email_template.txt matches the template
• files_unchanged - assets/sendmail_template.txt matches the template
• files_unchanged - assets/nf-core-airrflow_logo_light.png matches the template
• files_unchanged - docs/images/nf-core-airrflow_logo_light.png matches the template
• files_unchanged - docs/images/nf-core-airrflow_logo_dark.png matches the template
• files_unchanged - docs/README.md matches the template
• files_unchanged - .gitignore matches the template
• files_unchanged - .prettierignore matches the template
• actions_ci - '.github/workflows/ci.yml' is triggered on expected events
• actions_ci - '.github/workflows/ci.yml' checks minimum NF version
• actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
• actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
• actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
• readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
• readme - README Zenodo placeholder was replaced with DOI.
• pipeline_todos - No TODO strings found
• plugin_includes - No wrong validation plugin imports have been found
• pipeline_name_conventions - Name adheres to nf-core convention
• template_strings - Did not find any Jinja template strings (0 files)
• schema_lint - Schema lint passed
• schema_lint - Schema title + description lint passed
• schema_lint - Input mimetype lint passed: 'text/csv'
• schema_params - Schema matched params returned from nextflow config
• system_exit - No System.exit calls found
• actions_schema_validation - Workflow validation passed: linting.yml
• actions_schema_validation - Workflow validation passed: branch.yml
• actions_schema_validation - Workflow validation passed: fix-linting.yml
• actions_schema_validation - Workflow validation passed: release-announcements.yml
• actions_schema_validation - Workflow validation passed: awsfulltest.yml
• actions_schema_validation - Workflow validation passed: template_version_comment.yml
• actions_schema_validation - Workflow validation passed: download_pipeline.yml
• actions_schema_validation - Workflow validation passed: ci_immcantation.yml
• actions_schema_validation - Workflow validation passed: ci.yml
• actions_schema_validation - Workflow validation passed: clean-up.yml
• actions_schema_validation - Workflow validation passed: awstest.yml
• actions_schema_validation - Workflow validation passed: linting_comment.yml
• merge_markers - No merge markers found in pipeline files
• modules_json - Only installed modules found in modules.json
• multiqc_config - assets/multiqc_config.yml found and not ignored.
• multiqc_config - assets/multiqc_config.yml contains report_section_order
• multiqc_config - assets/multiqc_config.yml contains export_plots
• multiqc_config - assets/multiqc_config.yml contains report_comment
• multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
• multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
• modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
• base_config - conf/base.config found and not ignored.
• modules_config - conf/modules.config found and not ignored.
• modules_config - SAMPLESHEET_CHECK found in conf/modules.config and Nextflow scripts.
• modules_config - SAMPLESHEET_CHECK_ASSEMBLED found in conf/modules.config and Nextflow scripts.
• modules_config - FASTP found in conf/modules.config and Nextflow scripts.
• modules_config - GUNZIP_ found in conf/modules.config and Nextflow scripts.
• modules_config - VALIDATE_INPUT found in conf/modules.config and Nextflow scripts.
• modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
• modules_config - FASTQC_POSTASSEMBLY found in conf/modules.config and Nextflow scripts.
• modules_config - RENAME_FASTQ found in conf/modules.config and Nextflow scripts.
• modules_config - RENAME_FILE_ found in conf/modules.config and Nextflow scripts.
• modules_config - MERGE_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_FILTERSEQ found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_FILTERSEQ_POSTASSEMBLY_SANS_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_SCORE_UMI_R1 found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_SCORE_UMI_R2 found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_ALIGN_PRIMERS found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_ALIGN_CREGION found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_EXTRACT found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_ALIGN_UMI_R1 found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_ALIGN_UMI_R2 found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_EXTRACT_R1 found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_EXTRACT_R2 found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_SCORE_SANSUMI_FWD found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_SCORE_SANSUMI_REV found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_ALIGN_SANSUMI_FWD found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_MASKPRIMERS_ALIGN_SANSUMI_REV found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_PAIRSEQ found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_PAIRSEQ_ALIGN found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_PAIRSEQ_EXTRACT found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_CLUSTERSETS_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_PARSE_CLUSTER found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_PAIRSEQ_CLUSTERSETS found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_BUILDCONSENSUS_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_BUILDCONSENSUS_ALIGN_RACE found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_BUILDCONSENSUS_EXTRACT found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_POSTCONSENSUS_PAIRSEQ found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_ASSEMBLEPAIRS_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_ASSEMBLEPAIRS_SEQUENTIAL found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_ASSEMBLEPAIRS_SANS_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_PARSEHEADERS_COLLAPSE_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_PARSEHEADERS_PRIMERS_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_PARSEHEADERS_CREGION found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_PARSEHEADERS_PRIMERS_SANS_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_PARSEHEADERS_METADATA found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_COLLAPSESEQ_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_COLLAPSESEQ_ALIGN found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_COLLAPSESEQ_CREGION found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_COLLAPSESEQ_SANS_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_SPLITSEQ_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - PRESTO_SPLITSEQ_SANS_UMI found in conf/modules.config and Nextflow scripts.
• modules_config - FETCH_DATABASES found in conf/modules.config and Nextflow scripts.
• modules_config - UNZIP_DB found in conf/modules.config and Nextflow scripts.
• modules_config - CHANGEO_CONVERTDB_FASTA_FROM_AIRR found in conf/modules.config and Nextflow scripts.
• modules_config - CHANGEO_ASSIGNGENES found in conf/modules.config and Nextflow scripts.
• modules_config - CHANGEO_MAKEDB found in conf/modules.config and Nextflow scripts.
• modules_config - FILTER_QUALITY found in conf/modules.config and Nextflow scripts.
• modules_config - CHANGEO_PARSEDB_SELECT_LOCUS found in conf/modules.config and Nextflow scripts.
• modules_config - CHANGEO_PARSEDB_SPLIT found in conf/modules.config and Nextflow scripts.
• modules_config - FILTER_JUNCTION_MOD3 found in conf/modules.config and Nextflow scripts.
• modules_config - ADD_META_TO_TAB found in conf/modules.config and Nextflow scripts.
• modules_config - CHANGEO_CREATEGERMLINES found in conf/modules.config and Nextflow scripts.
• modules_config - REMOVE_CHIMERIC found in conf/modules.config and Nextflow scripts.
• modules_config - DETECT_CONTAMINATION found in conf/modules.config and Nextflow scripts.
• modules_config - COLLAPSE_DUPLICATES found in conf/modules.config and Nextflow scripts.
• modules_config - SINGLE_CELL_QC found in conf/modules.config and Nextflow scripts.
• modules_config - FIND_CLONAL_THRESHOLD found in conf/modules.config and Nextflow scripts.
• modules_config - REPORT_THRESHOLD found in conf/modules.config and Nextflow scripts.
• modules_config - DEFINE_CLONES_COMPUTE found in conf/modules.config and Nextflow scripts.
• modules_config - DEFINE_CLONES_REPORT found in conf/modules.config and Nextflow scripts.
• modules_config - DOWSER_LINEAGES found in conf/modules.config and Nextflow scripts.
• modules_config - AIRRFLOW_REPORT found in conf/modules.config and Nextflow scripts.
• modules_config - PARSE_LOGS found in conf/modules.config and Nextflow scripts.
• modules_config - REPORT_FILE_SIZE found in conf/modules.config and Nextflow scripts.
• modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
• nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
• nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.2

Run details

• nf-core/tools version 3.0.2
• Run at 2024-10-19 16:55:40