So it sounds like you're concerned about losing true signal by looking at too large a group of genes. My concern would be false positives/negatives based on the arbitrary nature of the cutoffs. I'd still argue in favor of GSEA over binning the genes by log2FC cutoffs.

In your example case you have 50/90 genes from ontology A that are between 1 and 1.5-fold upregulated. If this were the case then both your modified GO term enrichment and GSEA would find significance. A better question would be if you're still interested in this result if the other 40 genes are between 1 and 1.5-fold downregulated. From the method you've described, an ontology where genes are both up and down would be found significant in both analyses, whereas with GSEA it would not be significant.

One work around would be to filter out gene ontologies that are both up- and down-regulated, but is that fair? If you have an ontology that is 5-10 fold upregulated, and 0.5-1.0 fold downregulated, would you call that ontology upregulated or unchanged? GSEA would probably call that upregulated, but then what cutoffs are appropriate? What about 20-fold upregulated and 5-fold downregulated? Is that significant/meaningful?

I think you're asking an interesting question regarding ontologies within fold change intervals, I'd just be worried that the arbitrary nature of the cutoffs can lead to some false positives/negatives. If this is simply an in-house analysis for filtering or discovery work, then I don't see much harm. But if you're hoping to publish this analysis or downstream findings it might be difficult to defend this method to reviewers. GSEA using gene ontologies has its own drawbacks, but it's well established and accepted in the field, I can't imagine a reviewer giving it too much difficulty.