There's a bit of a misunderstanding here regarding what specifying multiple coefficients is actually testing. I have to admit that this is not terribly well spelled out in help(glmFit), although if you know how an LRT works and realize that you're feeding the coef vector to a function called glmLRT you can likely guess. To correct the misunderstanding, see the actual relevant code here.
In short, specifying coef=c(3:5) is not testing for a logical AND or an OR of DE genes in those coefficients. Rather, it's testing for genes for which the combined effect of c(3:5) is significant. In other words, it doesn't have to be the case that any individual coefficient is significant, just their combination. If you really want an AND operation, then test each of the coefficients, choose a more lax adjusted p-value cutoff and intersect the results. The equivalent for an OR operation would be the union of the results (you wouldn't need to loosen the p-value cutoff in that case).
As an aside, making c(3:5) a single group will produce different results still, since if those groups are very different from each other then the resulting coefficient will have large variance and be less likely to be picked up as significant.
By any chance, are you talking about interaction terms?
Not interaction terms. For instance, you treat a cell line with a compound of concentration 100nM, 200nM, 500nM. You want to find out if a gene 1) responds to any of the compound treatment; 2) responds to all of the compound treatment.
Does this make sense to you?
so it seems like you are expecting to make contrast specific design. In that case if you have already properly created your design matrix then it is not a problem to do the contrast specific DE test with edgeR and also do the entire estimation as well. Take a look at the links below.
link1
If you start looking at the edgeR manual from page 29, you will see that the
exactTest()can make the classical pairwise test between your groups of interest If you intend to use the glm based approach then the successive tests are pretty well defined in the manual of edgeR from page 30 and how to set the contrast between particular groups as in case 3 vs 4,, 4 vs 5 , etc The key is to prepare themodel.matrix()properly with the proper design of groups/levels and use contrast specific levels with glm-based approach. It should not be a problem to tackle it.I can only say this much unless I see the design of your model matrix , it is difficult to address about the specific contrasts that you intend to set.
This link is pretty informative as well but I will still advice for manual