I think it's pretty clear, that the used testset is from a segemehl publication, since we cited it several times. Nevertheless, I think the benchmark, as it is done by the segemehl developers, is quite good.

But you are right, there are several points open for discussion.

1. Of course one can use less than 10 errors and no multiple mapping loci for benchmarking. These values were just set for this benchmark, because one has to make a decision. But as we stated on the website, if one changes these numbers, the results will look different. We think that multiple mappings are important and should not be discarded, but for some experiments, one might not need them. For DE it is good to only use uniquely mapped reads, but for transcriptome reconstruction they might be useful. It is always a question of what one wants to do.
2. All the exact parameters can be looked up in the publication (link is on the website). For our test, we used the default parameter set, but other parameters were also tested in the publication. And you are absolutely right: If one changes the parameters, the results will directly change.
3. Absolutely correct! That's why we also plotted the memory consumption and the user time. This way people can take a look on the benchmark and make their decision on their own.
4. We are open to add other tools. Tophat is on our ToDo-list and will be added.

We just added this benchmark to our website, since a lot of people keep asking us what tool they should use and there is no simple answer to that. I think the way of how the segemehl people created their testsets is nice and a good starting point. But as you already mentioned, it is from the segemehl publication. We do not give any recommendations here. As you nicely pointed out, the memory consumption and the user time of segemehl is higher and thus it cannot be used by someone without a HPC. For these people, Bowtie2 or BWA is much better and they have a very good sensitivity.

The questions, one should ask oneself before using any mapping tool, are: What do I have (main memory, size of dataset, RNA-/DNA-Seq, etc.) and what is my goal (differential expression, transcriptome reconstruction, variant calling, etc.). The benchmark list can then give a hint of what tool might be useful. But it is far away from being complete.

My tip: First think about your data and your computer environment and then try to choose a reasonable mapping tool.

Point 3: That's why I like to have benchmarks. Actually, a researcher can use your questions, go to the benchmark page and see how the different tools behave. What's your remark here?

I completely understand your first argument and will double-check that question with the guys who created this benchmark and wrote the scripts.

Your second argument is also correct and I have to highlight here, that we do not want to give any recommendations. Segemehl performs best, but the difference in sensitivity is very small and there is the drawback of its memory consumption. I just think that this benchmark is just one possibility to get a feeling for mappers. I also know your benchmarks and they helped me a lot. This one is just another one, with a different point of view. And I'm pretty sure that hundreds will follow. In the end, the user has to decide which tool he or she wants to use.

The last point is a very good one and I really think that it would make sense to think about such a test. I'm not sure, how one would/should design it, since, as you said, there are strong connections between some mapping tools and downstream tools. But if Asaf has an idea, I'm open for any discussion for a benchmark using downstream analyses.

Following up your question about RazorS and splicing: Split-reads were not considered in this benchmark! The idea of using a mRNA-Seq dataset for this benchmark was the higher amount of errors in mRNA-Seq reads (even without split-reads). Post-transcriptional modifications, amplification errors, etc. result in reads which are harder to map.

From the supplement of the paper: 'For both Illumina mRNA-seq datasets, the post- processing involved removing reads that possibly overlapped exon-exon junctions. To achieve this, the entire dataset was mapped using segemehl (with -S option), STAR [3], TopHat2 [4], and SOAPsplice [5], and reads which were split-mapped by any of these tools were removed prior to down-sampling. In case of the paired-end mRNA-seq dataset, only paired-end reads were kept where both ends were not split-mapped by any of the tools.'

Split-read mapper were analyzed in another publication:

Hoffmann S, Otto C, Doose G, Tanzer A, Langenberger D, Christ S, Kunz M, Holdt L, Teupser D, Hackermüeller J, Stadler PF. A multi-split mapping algorithm for circular RNA, splicing, trans-splicing, and fusion detection. Genome Biol. 2014 Feb 10;15(2):R34.

Figure: Performance of various read aligners on simulated data sets with different splice events. For simulated 454 reads (400 bp), segemehl performed significantly better in detecting conventional and 'non-conventional' (strand-reversing, long-range) splice junctions. segemehl was the only tool that consistently recalled more than 90% of conventional splice junctions. For 'non-conventional' splice events, segemehl extended its lead to 40% for recall without losing precision. Likewise, compared to three of the seven alternative tools, segemehl had a 30% increase in recall for irregularly spliced Illumina reads (100 bp). Compared to TopHat2, it had a slight increase while reporting significantly fewer false positives. At the same time, segemehl's performance with simulated, regularly spliced Illumina reads was comparable with the other seven tools tested. gs, GSNAP; ms, MapSplice; ru, RUM; se, segemehl; sm, SpliceMap; so, SOAPsplice; st, STAR; to, TopHat2.

Rather than using RazerS 3, or any other tool for evaluating mappings that is unaware of the actual truth, I recommend using reads tagged by their genomic origins. Any other approach is biased, and inherently incorrect. For example, consider a 100bp read with 100 substitutions that coincidentally make it perfectly align to somewhere else in a genome. Using an edit-distance based aligner to determine the correct mapping, you will obviously get the wrong answer. Nothing can get the RIGHT answer, of course, since all initial information has been lost; and as such, awarding any points to any algorithm in that case demonstrates faulty grading. Similarly, any grading metric based on prediction rather than truth - even if some information content is retained in the data, unlike my example - cannot yield a fair answer, due to bias.

An unbiased approach:

Generate synthetic reads from random locations in the reference genome. Randomly mutate each read in specified ways, retaining the original genomic position of each base, and ideally, reflect error-mode mutations in quality scores. Grade mapping programs on whether they correctly align read bases to their original reference bases.

Any other approach is inferior in determining absolute mapping accuracy (though as lh3 mentions, mapping accuracy and the quality of end results are not equivalent). Specifically, an edit-distance-based approach, particularly with RNA-seq data (as in this test), will always give incorrect results for reads spanning introns sufficiently greater than read length; thus, grading such reads based on edit distance will consider an edit-distance-based aligner to be correct and a splice-aware aligner to be wrong, when the opposite is true. Edit distance is a useful heuristic for guessing at truth, but should never be used to quantify correctness; and particularly, should never be presented as truth, because it is not.

Well, question one has to be answered by yourself. Using multithreading will assure that you don't have to wait for 100 hours. And the last question... well, try it! :)