New module: checkqc #4158

matrulda · 2023-10-18T12:37:47Z

Adding new module checkQC. Related to issue: #4011

Depends on nf-core/test-datasets#1024 for tests to work.

Note: The CheckQC version used in the module is not the latest. I was forced to use an old one since there is an issue with the biocontainers for more recent ones. I aim to fix this and make a PR to upgrade to that version later.

PR checklist

Closes #4011

modules/nf-core/checkqc/tests/main.nf.test

sateeshperi · 2023-11-08T00:25:35Z

could you work on the failing tests and reach us out on nf-core/nf-test slack channel if you need help. cheers

matrulda · 2023-11-08T06:31:35Z

Yes, will look into that, thank you!

…en staging

* Code refactoring for mitohifi input * Fix md5sum * Fix md5sum for versions * Bring back md5sum value * Update md5sum again * Remove md5sum * Bring back versions.yml * It looks like $HOME needs be writable in the container. Trying mounting it * Add test for reads * Update modules/nf-core/mitohifi/mitohifi/main.nf Refactor the mitohifi command line Co-authored-by: Solenne Correard <[email protected]> * Fix md5sums --------- Co-authored-by: Ksenia Krasheninnikova <[email protected]> Co-authored-by: Matthieu Muffato <[email protected]> Co-authored-by: Solenne Correard <[email protected]>

* Adding module sentieon/tnfilter * WIP: CI-tests for sentieon/tnfilter. (test-data just local for now.) * Skipping conda-based test of sentieon/tnfilter * Remove avail_mem stuff. * Fix description of output vcf * Using test-data from nf-core test-datasets repo * add tests for sentieon/tnfilter

* nf-test for pycoqc. snapshots for versions and json['All reads'] * tags.yml * test naming conventions. Fix indentation. Add test for html output --------- Co-authored-by: Jose Espinosa-Carrasco <[email protected]>

* update bclconvert to 4.2.4 * regenerate snapshot

* ci(renovate): Change schedule to monthly and pinDigests https://emmer.dev/blog/keep-docker-base-images-updated-with-renovate/#default-renovate-config * ci(renovate): Add security PRs immediately https://emmer.dev/blog/keep-docker-base-images-updated-with-renovate/#custom-renovate-config * ci(renovate): Just use best-practices https://docs.renovatebot.com/presets-config/#configbest-practices * ci(renovate): Set PR hourly limit to 3 * ci(renovate): Add conda support Ulthran/ShotgunUnifrac#22 * ci(renovate): Add pre-commit * ci(renovate): Request modules instead of maintainers * ci(renovate): Don't scan the meta.ymls

…a to v0.27.0 (nf-core#4256) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…nf-core#4257) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* new module picard/scatterintervalsbyns * update annotsv to 3.3.6 * update installannotations * fix all modules conda declaration

chore(config): migrate config .github/renovate.json5 Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

nf-core#4220) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…f-core#4231) * new module picard/scatterintervalsbyns * update annotsv to 3.3.6 * update installannotations * try to fix linting with nf-test * small change to trigger tests * fix if statement * also test pytest * remove nf-test exclude from linting * try for a conda fix * revert last commit * Update .github/workflows/test.yml Co-authored-by: Adam Talbot <[email protected]> * temporarily reactivate pytest for vardictjava * run fastp test * revert test triggers --------- Co-authored-by: Adam Talbot <[email protected]>

* new module picard/scatterintervalsbyns * update annotsv to 3.3.6 * update installannotations * nf-test rtgtools rocplot * remove pytest tag * update according to comments * Update modules/nf-core/rtgtools/rocplot/tests/main.nf.test Co-authored-by: Adrien Coulier <[email protected]> * Update modules/nf-core/rtgtools/rocplot/tests/main.nf.test * update conda declaration --------- Co-authored-by: Adrien Coulier <[email protected]>

Emit crai file for CRAM file

…re#4214) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* We should use "error" rather than "exit 1" * Fixed the syntax * Removed the quay.io prefix since it is the default registry

…4259) * chore(deps): update biocontainers/goat docker tag to v0.2.5 * Version bump (across all 3 package systems) * Fixed the type of the "taxon" parameter * Updated the test md5 * Added me as a maintainer since GoaT is (partly) in my team --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Matthieu Muffato <[email protected]>

…se (nf-core#4245) change ndb to nin Co-authored-by: Damon-Lee Pointon <[email protected]>

* add nf-tests and snapshots for kallisto index & quant * rm kallisto tags from pytest_modules.yml * Update modules/nf-core/kallisto/index/tests/main.nf.test Co-authored-by: Adam Talbot <[email protected]> * Update modules/nf-core/kallisto/quant/tests/main.nf.test Co-authored-by: Adam Talbot <[email protected]> * Update modules/nf-core/kallisto/quant/tests/main.nf.test Co-authored-by: Adam Talbot <[email protected]> * update kallisto quant * Update modules/nf-core/kallisto/index/tests/main.nf.test Co-authored-by: Adam Talbot <[email protected]> * Update modules/nf-core/kallisto/quant/tests/main.nf.test Co-authored-by: Adam Talbot <[email protected]> * Update modules/nf-core/kallisto/quant/tests/main.nf.test Co-authored-by: Adam Talbot <[email protected]> * Update modules/nf-core/kallisto/quant/tests/main.nf.test Co-authored-by: Adam Talbot <[email protected]> * remove superfluous quote marks * fix linting --------- Co-authored-by: Adam Talbot <[email protected]> Co-authored-by: mashehu <[email protected]>

* Add Kmer coverage to module output * Update test.yml * Update modules/nf-core/genescopefk/meta.yml Co-authored-by: Matthieu Muffato <[email protected]> * Add bash directive * Fix whitespace --------- Co-authored-by: Matthieu Muffato <[email protected]>

fmalmeida

Hi @atrigila ,
Instead of removing the conda config, if the bioconda installation is being unreliable, one options would be to also use pip here as in the containers so to also have it:

name: checkqc
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - python=3.9 # whathever version is needed
  - pip
  - pip:
    - custom-pkg1
    - custom-pkg2

Would be worthy to check in the nf-core slack channel if this would be accepted or if other solutions would be proposed.

modules/nf-core/checkqc/tests/main.nf.test

atrigila · 2024-07-18T14:49:51Z

@nf-core-bot fix linting pretty please

atrigila · 2024-07-18T20:36:03Z

@fmalmeida adding the environment.yml with the version results in linting failure. This is related to this issue.

The environment.yml of the module checkqc is not valid: {'pip': ['checkqc==3.8.0', 'interop==1.2.4']} is not of type 'string'. Check the entry for dependencies.

This was observed in this module as well.

In addition, even when removing versions, the conda tests do not pass due to libmamba being used in the test.yml:

│   Test [a030344b] 'homo_sapiens illumina test_interop_bcl2fastqstats_flowcell - QC OK' FAILED (16.366s)                                            │
│                                                                                                                                                    │
│   Assertion failed:                                                                                                                                │
│                                                                                                                                                    │
│   2 of 2 assertions failed                                                                                                                         │
│                                                                                                                                                    │
│   Nextflow stdout:                                                                                                                                 │
│                                                                                                                                                    │
│   ERROR ~ Error executing process > 'UNTAR (1)'                                                                                                    │
│                                                                                                                                                    │
│   Caused by:                                                                                                                                       │
│     Failed to create Conda environment                                                                                                             │
│       command: conda env create --prefix                                                                                                           │
│ /workspace/modules/.nf-test/tests/a030344b93d4d35baf2381e4cf0f249f/work/conda/untar-6ec82032d38a28f610b86307ce4a0ff3 --file                        │
│ /workspace/modules/./modules/nf-core/checkqc/tests/../../untar/environment.yml                                                                     │
│       status : 1                                                                                                                                   │
│       message:                                                                                                                                     │
│         Error while loading conda entry point: conda-libmamba-solver (libarchive.so.19: cannot open shared object file: No such file or directory) │
│         Retrieving notices: ...working... done                                                                                                     │
│                                                                                                                                                    │
│         CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic                 │

fmalmeida · 2024-07-19T09:31:35Z

@fmalmeida adding the environment.yml with the version results in linting failure. This is related to this issue.

The environment.yml of the module checkqc is not valid: {'pip': ['checkqc==3.8.0', 'interop==1.2.4']} is not of type 'string'. Check the entry for dependencies.

This was observed in this module as well.

In addition, even when removing versions, the conda tests do not pass due to libmamba being used in the test.yml:

│   Test [a030344b] 'homo_sapiens illumina test_interop_bcl2fastqstats_flowcell - QC OK' FAILED (16.366s)                                            │
│                                                                                                                                                    │
│   Assertion failed:                                                                                                                                │
│                                                                                                                                                    │
│   2 of 2 assertions failed                                                                                                                         │
│                                                                                                                                                    │
│   Nextflow stdout:                                                                                                                                 │
│                                                                                                                                                    │
│   ERROR ~ Error executing process > 'UNTAR (1)'                                                                                                    │
│                                                                                                                                                    │
│   Caused by:                                                                                                                                       │
│     Failed to create Conda environment                                                                                                             │
│       command: conda env create --prefix                                                                                                           │
│ /workspace/modules/.nf-test/tests/a030344b93d4d35baf2381e4cf0f249f/work/conda/untar-6ec82032d38a28f610b86307ce4a0ff3 --file                        │
│ /workspace/modules/./modules/nf-core/checkqc/tests/../../untar/environment.yml                                                                     │
│       status : 1                                                                                                                                   │
│       message:                                                                                                                                     │
│         Error while loading conda entry point: conda-libmamba-solver (libarchive.so.19: cannot open shared object file: No such file or directory) │
│         Retrieving notices: ...working... done                                                                                                     │
│                                                                                                                                                    │
│         CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic                 │

Hi @atrigila ,

I see. Then unfortunately I think the best is to keep the env.yml file using the bioconda installation, however, using the same version that is used in the container. And then, just remove the conda testing profile from the CI, since the container executions work.

atrigila · 2024-07-19T12:47:22Z

@fmalmeida I think I prefer to stick to the current version as it specifically states that it is installed from pip. Since there is at least other module that has this same linting issue, and I think this will be addressed in the future, I would ask core-members @edmundmiller and @FloWuenne:

Can you disable the conda CI .yml for checkqc? As we do not have permissions to do so.
Could you allow us to merge with the linting issue (the same observed in module spotiflow?
Thank you!

disable in nf-test not in pytest

.github/workflows/test.yml

maxulysse · 2024-07-22T13:04:27Z

conda CI is properly disabled.
It was actually disabled for pytest, and it should have been disabled for nf-test actually.

matrulda requested a review from a team as a code owner October 18, 2023 12:37

matrulda requested a review from sateeshperi October 18, 2023 12:37

matrulda force-pushed the checkqc branch from fc374db to 808885c Compare October 18, 2023 12:39

sateeshperi reviewed Nov 8, 2023

View reviewed changes

matrulda and others added 24 commits November 8, 2023 08:34

Add template module with nf-core with some modifications

438d3b4

checkqc: Removed TODOs, added tests (need improvement), copy files wh…

7b2dbbf

…en staging

Use nf-tests for checkQC

90538df

pycoqc: add nf-test (nf-core#4238)

d2c747d

* nf-test for pycoqc. snapshots for versions and json['All reads'] * tags.yml * test naming conventions. Fix indentation. Add test for html output --------- Co-authored-by: Jose Espinosa-Carrasco <[email protected]>

Update seqkit/fx2tab to version 2.5.1 (nf-core#4240)

7c4ffb4

FEAT: update bclconvert to 4.2.4 (nf-core#4253)

5de8431

* update bclconvert to 4.2.4 * regenerate snapshot

chore(deps): update pre-commit hook python-jsonschema/check-jsonschem…

67aeaf5

…a to v0.27.0 (nf-core#4256) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

chore(deps): update pre-commit hook pre-commit/mirrors-prettier to v3 (…

a97fbad

…nf-core#4257) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Fix conda declaration (nf-core#4252)

26954e9

* new module picard/scatterintervalsbyns * update annotsv to 3.3.6 * update installannotations * fix all modules conda declaration

chore(config): migrate renovate config (nf-core#4255)

3fd1004

chore(config): migrate config .github/renovate.json5 Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

chore(deps): update docker.io/continuumio/miniconda3 docker tag to v22 (

7a57477

nf-core#4220) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

samtools/merge - emit crai file when cram file created (nf-core#4258)

d307124

Emit crai file for CRAM file

chore(deps): update biocontainers/chromap docker tag to v0.2.5 (nf-co…

1e5d77c

…re#4214) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Use "error" rather than "exit 1", and removed quay.io/ (nf-core#4248)

c126028

* We should use "error" rather than "exit 1" * Fixed the syntax * Removed the quay.io prefix since it is the default registry

change database sufix to nin which is more common in v5 and v4 databa…

10d6fa4

…se (nf-core#4245) change ndb to nin Co-authored-by: Damon-Lee Pointon <[email protected]>

nf-test interproscan (nf-core#4269)

d133bca

atrigila removed request for SPPearce, phue, muffato, maxibor, thanhleviet, lbeltrame and mashehu July 17, 2024 17:19

atrigila and others added 5 commits July 17, 2024 17:42

remove conda config

782454a

add metamap

d30a189

update test and snapshot

7eaf61a

Merge branch 'master' into checkqc

fae13f3

Merge branch 'master' into checkqc

48da3f9

fmalmeida requested changes Jul 18, 2024

View reviewed changes

modules/nf-core/checkqc/tests/main.nf.test Outdated Show resolved Hide resolved

atrigila and others added 2 commits July 18, 2024 14:27

add yml for tracking versions

5b584d8

skip checkqc conda profile

70f9b8f

fix linting

0d07bfd

Update test.yml

d9b2e91

disable in nf-test not in pytest

maxulysse reviewed Jul 22, 2024

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

maxulysse and others added 2 commits July 22, 2024 14:53

Update .github/workflows/test.yml

ba7bc1b

Merge branch 'master' into checkqc

d13ce00

atrigila added 3 commits July 22, 2024 14:37

Merge branch 'master' into checkqc

0e9516f

add fixme note as requested in slack

d2cc55d

Merge branch 'master' into checkqc

80bdbac

maxulysse merged commit 5848a1e into nf-core:master Jul 22, 2024
9 of 11 checks passed

atrigila mentioned this pull request Jul 29, 2024

Update checkqc to 4.0.3 #6051

Merged

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New module: checkqc #4158

New module: checkqc #4158

matrulda commented Oct 18, 2023 •

edited by atrigila

Loading

sateeshperi commented Nov 8, 2023

matrulda commented Nov 8, 2023

fmalmeida left a comment •

edited

Loading

atrigila commented Jul 18, 2024 •

edited

Loading

atrigila commented Jul 18, 2024

fmalmeida commented Jul 19, 2024 •

edited

Loading

atrigila commented Jul 19, 2024

maxulysse commented Jul 22, 2024

New module: checkqc #4158

New module: checkqc #4158

Conversation

matrulda commented Oct 18, 2023 • edited by atrigila Loading

PR checklist

sateeshperi commented Nov 8, 2023

matrulda commented Nov 8, 2023

fmalmeida left a comment • edited Loading

Choose a reason for hiding this comment

atrigila commented Jul 18, 2024 • edited Loading

atrigila commented Jul 18, 2024

fmalmeida commented Jul 19, 2024 • edited Loading

atrigila commented Jul 19, 2024

maxulysse commented Jul 22, 2024

matrulda commented Oct 18, 2023 •

edited by atrigila

Loading

fmalmeida left a comment •

edited

Loading

atrigila commented Jul 18, 2024 •

edited

Loading

fmalmeida commented Jul 19, 2024 •

edited

Loading