GSEA works by ranking all genes used in an analysis by some statistic. So this might be all genes in the genome (traditionally all genes with a probe on an array).

Start with an enrichment score of 0. Then take the gene ranked No. 1, and ask if it is in your gene-set. If it is you increase the enrichment score, if not decrease the enrichment score. Then move on to the gene ranked No. 2 and do the same thing. And so on down the whole list of genes. When you've done, find the highest the enrichment score you ever got. This is your Enrichment Score for that gene-set.

Next randomly shuffle the gene ranks (or sample labels) to put them in a random order, and repeat the scoring process above. Do this 1000 times.

Two things are now calculated - The P-value is calculated by looking at how often the Enrichment score from actual ranking is bigger than that for the random permulations. The Normalised Enrichment Score is calcualted by dividing the Enrichment Score from the actual ranking by the mean of the random permutations.

The FDR is calculated by comparing the distribution of Normalised Enrichment Scores from many different genesets.

Hi, I find this explanation very helpfull:

If you are only interested in enrichment alone, you chose the one sided test. But if you are additionally interested in underrepresentation, you choose the two-sided one. I cant see the data, but your task sounds like you are only interested in whether there is enrichment of genes of one gene set within another gene set, so you could do the one-sided one.