Hello Biostars community,
I need to transfer *.fastq.gz files to a server, but unfortunately my bandwidth is an important bottleneck in this process. I have heard that using rsync with the -z flag or scp with the -C flag compresses the data that are being transferred, thus reducing the transfer time. However, how effective is this when the files are already compressed? I also had heard that using rsync over ssh is much faster than scp. In that case, is compression still necessary? Is there are another approach you would recommend to speed up the transfer of *.fastq.gz files?
Thank you in advance for your time.
•
written
by
ropolocan
1 answer
You can't get much more compression out of compressed data. Otherwise you could keep running compression algorithms on different data until there's nothing left, and you can't extract back to the original data that way.
Are you running rsync without encryption, i.e. not over SSH? Encrypting data (as scp will do) will slow down your data exchange. Compressing already compressed data will further waste time. You might try parallelizing your data accesses, trying two or more simultaneous rsync calls from one or more computers to try to saturate your network connection. Once you have all the files locally, you can collect them together.
Log in to answer this question.
-
mv Question during kraken2-build (building database)
written by 10mz1 •
-
Can you concatenate fastq.gz (zipped) and fastq (non-zipped) files?
written by Eliveri •
-
concatenate 24 stranded specific RNA seq fastq libraries in linux ?
written by slin023 •
-
IGV over SSH?
written by Richard
-
hmmsearch vs hmmscan - e-value, speed, & output differences?
written by jabaron.phd •
-
what is the fastest way to extract reads of a specific barcode from a fastq
written by b10hazard •
-
Forum: The struggle between fastq and fastq.gz, compressed v/s uncompressed file formats
written by lakhujanivijay
-
Tool: Introducing Clumpify: Create 30% Smaller, Faster Gzipped Fastq Files. And remove duplicates.
written by Brian Bushnell
-
An Alternative To Rsync (Or Useful Parameters)
written by Stefano Berri
-
Indexing Compressed Fastq-Like Data
written by 2184687-1231-83-
