This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Send BAM files from server to cluster

Hi,

I have 82 BAM files + BAI in a remote directory which I have acess via smb I want to send these files to server which I have acess via ssh from this remote directory. So from my computer (SMB) ------> lab files (SSH) ------> Cluster.

I have tried using SCP with and without the option -C and it the transfer ends before 100% The File is sent yet only 50/60% of the file.

I know this is not bioinformatics question directly yet having files in order for variant calling and other pipelines is crucial. Is there any better way to transfer over BAM files? What is the best practice in this situation?

Thanks.

bam
• 1,541 views • link
updated by ATpoint ATpoint • written by Manuel Sokolov Ravasqueira • Manuel Sokolov Ravasqueira

2 answers

Seconding both GenoMax and Zhenyu Zhang. rsync has the advantage that it has many checks in place to guarantee file integrity. It will only create the final visible file on the destination once upload finished properly. So you do not run into the relatively common scp issue that you have a file sitting at destination that you do not know whether it is complete.

Speaking of integrity: Run md5sum on the files at origin location, and then again at destination and compare them to ensure everything went properly.

• 0 views • link
written by ATpoint ATpoint

Sounds like your connections are not very reliable. I would recommend trying rsync .

• 0 views • link
written by GenoMax GenoMax

second GenoMax. scp only for single small file, rsync for large ones.

• 0 views • link
written by Zhenyu Zhang Zhenyu Zhang

Log in to answer this question.

More posts like this