Q: What is the relationship between library size and normalization factor?
The answer is right there if you read a bit further:
"Indeed, it is known that TMM normalization factors do not take into
account library sizes. This fact is illustrated in Figure 1 by an
almost horizontal regression line. On the contrary, RLE and MRN
factors are closer to each other, and share a positive correlation
with the library size."
Q: What does it mean if the regression line have R^2 of 0.9?
A regression (linear regression here) R2 tells how good the curve (here line) fits is to your data. If all the data are on line, R2 = 100. You can also think this in term of correlation. Correlation means "how good" one variable can be predicted from another variable. In fact, the goodness of fit R^2 is numerically equal to the square of Pearson correlation (rho).
R2 = 0.9
=> rho (Pearson correlation) = sqrt(0.9) = 0.94
By looking either of the numbers (R^2 or rho), you can conclude that there is a very good (linear) correlation among two variables and one can be almost perfectly predicted from other. By looking at the line (red or blue line, say), you can easily see that when one variable increases, the other too (in mathematical term, the slope of the line is +ve). This information is also conveyed by the sign (positive) of R^2.