This is a test version of Biostars. For the public version, visit https://www.biostars.org.
DiscoSNP++ 2.2.0 Huge number of temporary files

Hi,

I noticed the program created a lot of temporary files named trashme_* at one point I checked and there were 8000+ files that were open. Having this many files open seemed a little strange to me. Is this something intentional? Can there be a bug somewhere in the code (like forgetting to close files ?). If this is the intention, I am curios to understand the use of that many files.

Thanks

Best Regards

T. Hamdi Kitapci

discosnp
• 2,282 views • link
updated by Ram Ram • written by tkitapci tkitapci

1 answer

The trashme_* files are temporary partitions for the first step of DiscoSNP (k-mer counting). It is normal to have many of them (around a thousand), however 8000+ seems excessive. What were your command line parameters and the input data size?

• 0 views • link
updated by Ram Ram • written by Rayan Chikhi Rayan Chikhi

Hi,

The problem is not the number of files but the number of effective "open" files. I did not look at the source code but it seems odd to me that all 8000+ files are "open". I am wondering if keeping all those files in "open" state is the intention of the program or if the programmer forgot to close some of the files after the processing is completed. This is the output I get from my run:

https://docs.google.com/document/d/1jpooJySV1rKTQEronzGyprgdSISPwdhcbBJSDOflR_4/edit?usp=sharing

total input file size is 28 GB (2 fastq files each 14GB)

Thanks

Best Regards
T. Hamdi Kitapci

• 0 views • link
updated by Ram Ram • written by tkitapci tkitapci

Thanks. This is odd, it says here on the log that there are 880 partitions. Each partition indeed corresponds to a file that will be opened at the same at the others partition files. So there should be only 880 opened files (but I'm not fully familiar with the other graph creation steps, "debloom").

Can you let me know the complete prefix of those trashme* files? Should be trashme_[pid]_[informative prefix I'd like to know about].parts.[number]

• 0 views • link
written by Rayan Chikhi Rayan Chikhi

Log in to answer this question.

More posts like this