If you want to specifically check for under-enrichment, you can run a left-tailed one-sided Fisher's exact test. Here you're checking if the observed frequency is significantly smaller than the background frequency, i.e. you have less hits than you'd expect from random chance.

In R, you'd run fisher.test() with the parameter alternative="less" to specify the left tail.