This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Submit High-Throughput data to GEO. Problems with FTP connection

I open this question because I didn't find much information about this topic so far. I am trying to upload data from a RNA-seq project to GEO (387,5GB) with UNIX command line and I am getting the error:

 Lost data connection to remote host after 1xxxxxxxx bytes had been sent: Broken pipe.

The number of bytes being variable each time. After asking the IT service in my institution, they told me that the FTP protocol is very slow and the broken connection is expected for such big files.

I solved The issue using the scipt in this post

However, it will be helpful if anybody can share other answers to this problem, maybe also to improve the speed, as it is taking a lot of time to submit all files.

Thanks a lot!

ftp next-gen geo
• 6,799 views • link
updated by Arup Ghosh Arup Ghosh • written by mgdrnl • mgdrnl

That does not sound like a bioinformatics question to me!

• 1 views • link
written by lakhujanivijay lakhujanivijay

Submit your data to ArrayExpress, it has a better interface for metadata management and file uploading. The direct FTP connection is also fast, accession ids are provided within a couple of hours and with a week they will provide the reviewer account details.

• 1 views • link
written by Arup Ghosh Arup Ghosh

Thanks! I will definitely try ArrayExpress next time.

• 1 views • link
written by mgdrnl • mgdrnl

1 answer

Are you sure you want to submit RNA-seq data (raw data?) to GEO? You should submit to SRA instead. NCBI supports upload via Aspera connect (ascp with very similar command line interface to scp) which is faster and more robust against interrupted network connection. See : https://www.ncbi.nlm.nih.gov/books/NBK242625/ and https://www.ncbi.nlm.nih.gov/sra/docs/submit/

• 1 views • link
written by Michael Dondrup Michael Dondrup

Thank you very much for the recommendation! the truth is I didn't know about SRA. Unfortunately the journal I am submitting the paper asks me to deposit the data (raw data, fastq files) to GEO or ArrayExpress. I will try with ArrayExpress next time, as someone above said it is better.. unless the journal changes their policy.

• 1 views • link
written by mgdrnl • mgdrnl

Log in to answer this question.

More posts like this