Samtools: "Too Many Open Files"

Question

[Zhshqzyc]

Hello, I used samtools to sort a 210GB bam file. The command was:

samtools sort -on in.bam out.prefix

About 1058 .bam files were created. Each of them has about 200,000kb size. The names are

out.prefix.0000.bam
out.prefix.0001.bam
out.prefix.0002.bam
....
out.prefix.1057.bam

However after several days running, there was an error finally.

[bam_sort_core] merging from 1249 files...
open: Too many open files
[bam_merge_core] fail to open file out.prefix.1020.bam

Thanks for advice.

[Michael Schubert]

Hi zhshqzyc! You have asked 15 questions so far and not accepted a single answer. Please click on the outline of the checkmark next to it if it solves your problem.

[Zhshqzyc]

Sorry about it. I just realized there is such a function existing.

Answer 1

[Istvan Albert]

It is common to have a soft limit of 1024 open files by default. See what

ulimit -n

tells you. If it is 1024 then it is likely that your system allows more files to be open only that the ulimit imposes the per-process limit of 1024. See the manual page for ulimit on how to change this.

Answer 2

[brentp]

You can increase the argument for maxMem via -m. This will reduce the number of temporary files.

The default is 500000000 so try 10 times that value.