Hi All, I have a bit of a dilemma. I have to process a rather large BAM file into an interleaved FASTQ. After it's sorted and indexed, I use the samtools collate command as follows: samtools collate -u -o input_sorted_collated.bam input_sorted.bam
The size of input_sorted.bam is ~147GB and after a long execution the process (unsurprisingly) runs out of space on /tmp. Unfortunately, the collate command of samtools doesn't recognize the -T option to direct the temporary files elsewhere. I guess the developer hasn't considered the possibility of really large datasets.
Has anyone run into this issue before? Can you suggest a solution/hack/workaround? Any thoughts are very much appreciated!
Sasha
•
written
by
obuzko •
1 answer
I use the samtools collate command as follows:
samtools collate -u -o input_sorted_collated.bam input_sorted.bam
(not tested) use <prefix>
http://www.htslib.org/doc/samtools.html
samtools collate [options] in.sam|in.bam|in.cram [<prefix>]
<prefix>is optional. If <prefix> is absent, collate will write the temporary files to a system-dependent location (/tmp on UNIX).
Thank you so much Pierre, I was stuck here, but specifying the prefix although I'm outputting to stdout solved it (the node ran out of space on /tmp). The docs of collate aren't very clear here... they just say the prefix is necessary if the explicit output options aren't specified. Brr.
Log in to answer this question.
-
Error 137 in Epi2me TB-AMR Workflow
written by End •
-
Piping samtools to R
written by noodle
-
samtools collate
written by Sara •
-
Does running samtools sort in parallel result in unexpected output due to temporary file conflicts?
written by octpus616 •
-
Filtering a multi-person structural variant VCF file by type of SV
written by tacrolimus
-
how many files are generated after sorting of .bam file ?
written by iqra •
-
bam-readcount not finding reference in bam file
written by obuzko •
-
Aligning Ion torrent sequenced genome with a reference sequence
written by Paul
-
How to know if samtools sort exited normally [left temporary files behind]
written by Sukhi Singh
-
Blog: Building A Circular De Novo Assembler
written by Geneious •
The solution Pierre Lindenbaum provided should work. Always read the manual first.
Just as a comment, better avoid these kinds of statements especially when the issue is about really basic things such as memory usage. The
samtoolsdeveloper(s) know what they do, these tools are routinely used by thousands of people all year, and some larger files are not uncommon. Fitting entire files into memory is uncommon as not every usage has a heavy server node available. Typically, if one encounters standard issues such as memory problems one is doing something wrong. 1xx Gb files are not uncommon either, standard 30-50x short-read human WGS for example. Thousands of these samples exist at NCBI.