wget host address error

[mniederhuber]

Description of the bug

I tried out the dev branch and am encountering a wget error in process NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP
The underlying error is:

wget: unable to resolve host address 'ftp.sra.ebi.ac.uk'

Getting the error for a number of SRX experiment ids that have successfully downloaded with sra-tools in the past.

I'll try to see if I can figure out the issue, but figured I'd bring it up.

Command used and terminal output

#! /bin/bash
#SBATCH --mem=8G
#SBATCH -t 6:00:00
#SBATCH -p general
#SBATCH -o var/log/fetch-%j.out
#SBATCH -e var/log/fetch-%j.err

module load nextflow

nextflow -log var/log/.fetchngs run nf-core/fetchngs -r dev \
	-profile unc_longleaf \
	-params-file config/fetchngs_params.yaml

Relevant files

logfile.txt

System information

Nextflow 23.04.02
HPC
slurm
Singularity
RHEL8
fetchngs dev

[Midnighter]

Could be intermittent network or server issues. ENA/SRA do see a lot of traffic.

[CJPerkins1]

I'm experiencing the same issue with wget using the dev branch, were you able to get this to work?

[pvanheus]

This is because of a problem with the Singularity container. A certain generation of containers was built with a Busybox that had a broken /etc/resolv.conf. I have reported this to the Galaxy folks who build the Singularity containers and will follow up once that is fixed.

[josemunozc]

I think the problem is the container.

$ module load singularity-ce/4.1.0
$ singularity shell depot.galaxyproject.org-singularity-wget-1.20.1.img
WARNING: Skipping mount /var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
Singularity> wget -t 5 -nv -c -T 60 -O ERX2235404_ERR2179103_2.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/ERR217/003/ERR2179103/ERR2179103_2.fastq.gz
wget: unable to resolve host address 'ftp.sra.ebi.ac.uk'

However, if I try the latest version of the container (check https://depot.galaxyproject.org/singularity/):

$ singularity pull https://depot.galaxyproject.org/singularity/wget:1.21.4
$ singularity shell wget\:1.21.4
Singularity> wget -t 5 -nv -c -T 60 -O ERX2235404_ERR2179103_2.fastq.gz ftp.sra.ebi.ac.uk/vol1/fastq/ERR217/003/ERR2179103/ERR2179103_2.fastq.gz
Singularity> ls ERX2235404_ERR2179103_2.fastq.gz
ERX2235404_ERR2179103_2.fastq.gz

So, I guess the solution is to instruct Nextflow to fetch the latest image in modules/local/sra_fastq_ftp/main.nf:

    conda "conda-forge::wget=1.20.1"
    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        'https://depot.galaxyproject.org/singularity/wget:1.20.1' :
        'biocontainers/wget:1.20.1' }"

change to (conda also for consistency but I haven't test):

    conda "conda-forge::wget=1.21.4"
    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        'https://depot.galaxyproject.org/singularity/wget:1.21.4' :
        'biocontainers/wget:1.21.4' }"

[josemunozc]

sorry, just realized that the suggested change already made it to the dev branch :p

[maxibor]

I can confirm, updating the wget container to v.1.21.4 (with fe27569) fixed this issue.

[LKeene]

Hello! I am still getting this error even with the fix suggested above. The line before the wget error has a warning regarding my singularity. Could this be part of the problem?

Command error:
WARNING: Skipping mount /usr/local/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
wget: unable to resolve host address 'ftp.sra.ebi.ac.uk'

Thanks!

[IBEXCluster]

Dear @LKeene
I was able to get successful results when conda is used.
For example, I was using -profile conda

nextflow run nf-core/fetchngs -r 1.12.0 -profile conda --input ids.csv --outdir results_naga_test -c ibex.config

[nick-youngblut]

I was able to get successful results when conda is used.

A bummer that users are currently forced to use conda instead of Singularity, at least for now.