edgeR incorporates a great deal of advanced and novel statistical theory but it is based on generalized linear models (GLMs). GLMs are based on the idea of a variance function, which determines the mean-variance relationship of the response variable. If you have a good understanding of GLM theory, then you should be able to appreciate how edgeR's negative binomial (NB) dispersions enter into the regression coefficients and into the statistical tests. You don't need to know anything about NB GLMs specifically, because the principles and the estimation algorithm are the same for all GLMs. If you know GLM theory in general, and you know the NB variance function in particular, then you should be to work the rest out.
Briefly speaking,
- the NB dispersion determines the variance function,
- the variance function divides into the GLM working weights,
- the working weights enter into the reweighted linear regressions that determine the GLM regression fit.
Furthermore, the NB dispersion enters directly into the NB likelihood function and into the residual deviances that determine the test statistics for the regression coefficients once the GLM fits are done.
Here are some relevant references relating to empirical Bayes and to GLMs (all of which are cited in the edgeR User's Guide and function help pages).
Dunn & Smyth (2018) explain the theory of GLMs from scratch, and also briefly cover NB GLMs in particular.
If you don't already understand GLMs, then you are always going to have trouble understanding edgeR, because it is a very advanced tool that uses GLMs as the starting point.
Unbiased estimation of the NB dispersion (why MLE is wrong):
Robinson MD, Smyth GK (2008). Small sample estimation of negative binomial dispersion, with applications to SAGE data. Biostatistics 9, 321-332.
Empirical Bayes estimation of the NB dispersion using conditional likelihood (edgeR v1):
Robinson MD, Smyth GK (2007). Moderated statistical tests for assessing differences in tag abundance. Bioinformatics 23, 2881-2887.
Empirical Bayes estimation of the NB dispersion using generalized linear models (edgeR v2):
McCarthy DJ#, Chen Y#, Smyth GK (2012). Differential expression analysis of multifactor RNA-Seq experiments with respect to biological variation. Nucleic Acids Research 40, 4288-4297.
Empirical Bayes estimation of the NB quasi-dispersion (edgeR v3):
Lund SP, Nettleton D, McCarthy DJ, Smyth GK (2012). Detecting differential expression in RNA-sequence data using quasi-likelihood with shrunken dispersion estimates. Statistical Applications in Genetics and Molecular Biology Volume 11, Issue 5, Article 8.
Negative binomial generalized linear models:
Dunn PK, Smyth GK (2018). Generalized linear models with examples in R. Springer, New York, NY. doi:10.1007/978-1-4419-0118-7.
Have you read the most recent edgeR paper at https://doi.org/10.1093/nar/gkaf018, which gives a summary of the statistical methods in edgeR?
There is no parameter in edgeR called "alpha", so I wonder what you are reading.
Yes, the negative binomial dispersion parameter in edgeR is the squared CV of the gamma distribution.
The negative binomial dispersion does affect the regression estimates.
Maximum likelihood produces biased and highly uncertain dispersion estimates. It is completely unsuitable for dispersion estimation of omics data with small to moderate sample sizes.
edgeR versions 3 and 4 have moved to use global estimates of the negative binomial dispersion in favour of empirical Bayes quasi-dispersion estimation. See the summary in the above paper.
The edgeR User's Guide is primarily intended to guide users of the software, so it only gives a very brief summary of the mathematics behind edgeR. Details of the mathematics are more the domain of the published journal articles. Regression estimation in edgeR follows standard generalized linear model theory for the negative binomial family, so a textbook on glms would be relevant.