Solved and abstract will be submitted shortly; thanks for your courteous help and attention!
Original Content (added back by @RamRS after OP deleted it):
Biostars,
I will soon have a directory of about 10 Tb with approximately 180,000 .fna files representing the PATRIC database. I have a .txt file that is in list form, with each of the accession numbers that form the prefix for the .fna names for each file. eg-
573.14498
573.14499
573.145
file names correspond, viz-
573.14498.fna
573.14499.fna
573.145.fna
I would like to concatenate files so that they are approximately 1 Gb each. That means, I would like approximately 1000 multi fastas where there are now 180,000.
What is the best scripting or command strategy or utility to accomplish this task. I admit this is starting to take up more space than I have available on my RAID, so some simultaneous concatenation and compression would be desirable.
Best,
Bert Gold, PhD, FACMGG, CGMB San Mateo
1 answer
Are you sure you want / need to mirror the whole PATRIC database? And why compressing thousands of bundled fastas into common files? Wouldn't make it more difficult to retrieve the data from each particular fasta? And why 1Gb chunks? Wouldn't make more sense to at least bundle the files in a way to reflect the original PATRIC organization?
One approximate simpler solution to your question would be to concatenate the .fna files from each PATRIC folder into one big file, then compress this big file with bgzip (from HTSlib) and index the file with samtools faidx. This way, you save space and keep fast random access to the genomes.
That doesn't explain the absurd solution you are building here.
Log in to answer this question.
Apologies in advance for multiple questions but here goes: How will this data arrive? On external disk(s) or via download? What is the logic behind creating 1Gb file chunks (before or after compression)? What is the ultimate application for this data (e.g. as a reference, for doing assemblies)? You mention RAID but exactly what kind of hardware do you have access to?
To answer the actual question, you probably won't do much better than a shell script:
https://superuser.com/questions/884868/merge-small-files-to-larger-one-with-specific-size-and-preserving-lines
But like genomax I have more queries about the 'why' than the 'how'
Thank you for your kind and insightful questions and suggestions.
I would just work with the files you downloaded. Creating a blast index for 10T of data is likely not going to work with amount of RAM you have if you were worried about 10T filling your data RAID.
Thank you for the very insightful comment, jrj.healey
I still suggest you replace gzip with bgzip, as it will allow indexing the compressed fastas with
samtools faidxand rapid random access. bgzip-compressed files are just a bit larger than gzip-compressed files.Thank you for the opportunity to ask a question!
I''m trying to get takeaways from that:
Hi bgold04, please do not delete your questions after they have been answered. The purpose of this forum is two-fold: to provide immediate answers for questions, and to serve as a searchable repository of questions and answers. Deleting your question defeats the second purpose.
Do not go back and remove relevant content from your post after they're addressed. The forum is not your personal help desk, it is a knowledge repository. Please consider this your final warning, repeating such behavior will cause your account to be suspended/banned.