Please show code and plots, words alone cannot help debugging.

More details would be helpful, but it sounds like you are looking at the wrong comparison/contrast. For example, this is what I would see if I compare control to itself.

For example, my design compares condition to control. The control is represented by coefficient 1 and condition is coefficient 2. With the way I setup the design when I run the test and specify coef = 2 it will compare condition to control. If I compare control to itself by coef = 1 it is not proper and results in very low fold-changes

> table1 <- glmQLFTest(fit, coef = 2)$table
> table2 <- glmQLFTest(fit, coef = 1)$table
> head(table1)
                            logFC     logCPM            F      PValue
ENSMUSG00000033845.13  0.85170453 -0.9476147  3.161095975 0.081842447
ENSMUSG00000033813.15  1.63796058 -1.2257288 10.471165674 0.002216420
ENSMUSG00000025907.14 -0.04744762 -1.4859080  0.006494684 0.936108420
ENSMUSG00000090031.2   0.74307672  0.4415572  3.467670646 0.068798899
ENSMUSG00000051285.17  1.91045100 -2.0703469  8.740009379 0.004844126
ENSMUSG00000061024.8   1.45765352 -0.9993126  8.529882718 0.005339044
> head(table2)
                          logFC     logCPM        F       PValue
ENSMUSG00000033845.13 -21.44758 -0.9476147 693.1961 6.689046e-30
ENSMUSG00000033813.15 -22.11838 -1.2257288 684.3716 8.884856e-30
ENSMUSG00000025907.14 -21.98321 -1.4859080 569.5287 5.067023e-28
ENSMUSG00000090031.2  -20.18491  0.4415572 820.4644 1.565172e-31
ENSMUSG00000051285.17 -23.18911 -2.0703469 512.1814 5.112515e-27
ENSMUSG00000061024.8  -21.98225 -0.9993126 696.6826 5.984925e-30