Update: The site is down even though it was published in Nature Communications only 2 years ago. http://www.nature.com/articles/ncomms7275
I have just discovered the following tool: The GCAT benchmarking tool is a web-application with which one can compare software and pipelines for read alignment and variant calling. It provides a couple of challenge datasets and precomputed results for different variant calling pipelines, which can be compared based on plots of different statistics, eg. Venn-diagrams to depict overlap of variant calls. It also allows to test your own software and pipelines by uploading the results and doing a comparison with pre-computed pipelines.
From the web-site:
Have you ever wanted to benchmark genome analysis tools? With GCAT, it is now incredibly simple. Genome Comparison & Analytic Testing (GCAT) was developed as a platform to facilitate easy testing, comparison, and benchmarking of tools. Want to know how BWA + GATK Unified Genotyper fares against other analysis pipelines? Check out the report.
It appears to me that this kind of benchmarking application can be highly useful for evaluation of variant calling pipelines, and in order to establish a rational approach to choice of software and parameter settings.
