Assuming OP likely has a single ethernet port on their machine this sounds like an invitation for trouble. Multiple threads are just going to compete with each other and eventually may take more time to complete the download.
I want to download the following fastq files in Salmon all at once:
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/014/SRR10611214/SRR10611214_1.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/014/SRR10611214/SRR10611214_2.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/015/SRR10611215/SRR10611215_1.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/015/SRR10611215/SRR10611215_2.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/016/SRR10611216/SRR10611216_1.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/016/SRR10611216/SRR10611216_2.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/017/SRR10611217/SRR10611217_1.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR106/017/SRR10611217/SRR10611217_2.fastq.gz
Is there a way to do this using a bash for loop or fasterq-dump?
1 answer
Save the links in a text file and use GNU parallel with wget to download the files simultaneously.
parallel --verbose "wegt {}" ::: $(cat download.txt)
If you want to use multiple connections for download use aria2.
parallel --verbose "aria2c -x <number of conections> {}" ::: $(cat download.txt)
Log in to answer this question.
Is
wget -i your_listwhat you are looking for ?Put the links in a file (one per line). Then use
wget -i url.txt(if your file is calledurl.txt. If you want to usecurlinstead then usexargs –n 1 curl –O < url.txt.After making a input_files.txt file with only SRR names and put this in a input_file_folder. You can use this script: