This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Permutations FGSEA

Good afternoon,

I was wondering if someone could advise me on the following:

I am running FGSEA with my DESEQ2 results in Hallmarks following below code:

# Prepare results from DESEQ2

res<- res[order(-res$stat),]
ranks<-res$stat

names(ranks)<-res$hgnc_symbol

# Run gene set enrichment analysis 

fgseaRes <- fgsea(pathways=pathways.hallmark, stats=ranks,nperm=10000)

However, my pathway results change signficantly if I change the number of permutations (e.g. from 1000 to 10000). I am not quite sure which number of permutations I should choose?

Thank you!

fgsea
• 3,127 views • link
written by Nona • Nona

the smallest p-value for 1000 permutation is 1/1000 and if you increase it to 10000, the smallest p-value you can get is 1/10000

• 0 views • link
written by Ming Tommy Tang Ming Tommy Tang

2 answers

You don't really need to specify nperm anymore (it even produces a warning if you do so). It's only supported for backward compatibility purposes and it results in using an older and less efficient algorithm.

• 0 views • link
written by alserg alserg

Thank you for your reply. Do you know what number of permutations is applied then, if I do not specify it? Thanks again!

• 0 views • link
written by Nona • Nona

If you don't specify it, then fgseaMultilevel is used, which doesn't have nperm parameter at all and can calculate arbitrarily small p-values, not limited to 1/nperm.

• 0 views • link
written by alserg alserg

Thank you. Please see my comment below regarding the different results i get with 1000, 10000 and without permutations. Thanks a lot for your help!

• 0 views • link
written by Nona • Nona

I have run it with permutations = 1000, 10000 and not specify the permutations. The pathways are the same, but the p-values vary significantly. What can I trust??

enter image description here

• 0 views • link
written by Nona • Nona

True GSEA P-values are hard to calculate exactly, thus FGSEA (and other similar programs) _estimate_ them with a certain level of accuracy. Higher nperm gives you higher accuracy, but takes more time, fgseaMultilevel procedure (which is executed when you don't specify nperm) has a better accuracy to time tradeoff. Anyway, all of these methods are non-deterministic and can give you slightly different results each time you run it, so there can be inconsistencies near the selected significance threshold.

• 0 views • link
written by alserg alserg

Ah ok that explains why the results between "not specify permutation" and 100.000 permutations are similar. Thank you very much for your reply.

• 0 views • link
written by Nona • Nona

Log in to answer this question.

More posts like this