http://sourceforge.net/p/samtools/mailman/message/31544865/ :

When MarkDuplicates is accumulating information about mate pairs, it writes information to temporary files, one per reference sequence. If the reference has many sequences, there may be many temporary files. MAX_FILE_HANDLES_FOR_READ_ENDS_MAP controls how many of these files may be open at once. If this number is low, then the program needs to close and re-open files more frequently. This results in slower program execution, but the results should not change.