Bourgon et al. nicely showed that you can perform independent filtering to increase power in differential expression analyses (they were using microarrays, but the same holds for RNAseq). The general idea, as you seem to be correctly alluding, is that there's some expression level below which we simply lack sufficient power to even bother testing for significant DE. If you perform independent filtering (see the genefilter package in Bioconductor for some handy functions), then you'll end up weeding out those low expressing genes in a meaningful way. This should produce the better GO results you're after and have the added bonus of also yielding better DE results.
BTW, have a look at the camera() and roast() functions in the limma package for methods for gene-set testing that are better than the standard "just do a hypergeometric test" methods.
One last note is that you'll probably do the filtering on the whole dataset if you plan at the outset to make all two-way comparisons between 2 groups (I say "probably" because I don't know the details of your experiments), rather than performing each comparison completely separately.
