How to compare different chipseq peak calling methods?

Question

[jesvinjoy35]

Hi, I have been using various peak calling methods like MACS2, JAMM, BinQuasi and Zerone across the same samples for transcription factors and for histone modification. JAMM works well with and without control samples and gives the highest number of significant peaks while all the other methods work well with the samples with control and give a significant no. of peaks. Now, I want to know which of these methods is the best for peak calling. Is there a way to compare these methods? I read in a few papers to use Encode data as a reference but how to compare it with the peaks that I obtain from these methods?

Answer 1

[jared.andrews07]

Really depends what you mean by "best", which only you can really define for your purposes. Is best the one that's the most sensitive or the one that's the most accurate? For which factor? Some callers are better for broad marks (especially histone modifications) than others.

You're probably best off trying to find a paper that compares them on data similar to your own and making your choice based on those results rather than doing an objective comparison yourself.

[jesvinjoy35]

@jared.andrews07 by best I mean the most accurate ones. It doesn't matter for which factor. The tools I tested works well with both narrow and broad peaks. So, I would like to know if there is any way to calculate sensitivity, specificity and accuracy for each of these tools. I did try finding some papers but I couldn't understand how they incorporate Encode data as a reference. Please suggest

[jared.andrews07]

Well, factor really does matter, considering some peak callers perform better for narrow or broad peaks than others. Lots of papers (like this one) have compared dozens of peak callers to find those that are most accurate, specific, and sensitive. Some use ENCODE data, some don't, not that it really matters.

Most identify a handful that are all within a few percent of each other depending on the evaluation metric, occasionally noting that specific algorithms are significantly better for certain situations than others. I doubt you want to go through the effort of doing such a comparison on your own data. There are lots of suitable peak callers - pick one (or multiple) that fit your situation. You can always overlap peaks from multiple callers to get more high confidence regions.