Yes, I agree that the SVA approach is a great tool for identifying (and correcting for) latent sources of variation. Moreover, identified SVs could probably be also input into variancePartition to explore how they explain variation in comparison to known covariates, and that would surely be of interest!
Nonetheless, this post was more addressed to the more general, "unsupervised", exploration of the data. I've used SVA to great results, but one could argue that sometimes "protecting" the comparison/phenotype of interest is a bit "supervised" in the sense that you're intentionally avoiding variables with some correlation to the phenotype of interest. Sometimes one may want to include known and measured covariates if their effect is clear, even at the cost of losing some of the biological signal because of them being somewhat associated to the phenotype of interest.

Search for feature selection in machine learning. Some approaches such as lasso or tree-based methods (e.g. random forest, xgboost) output a variable importance. Another popular approach is recursive feature selection.
Thanks! I thought feature selection methods were generally applied to choose/collapse features that make up the info in the data (e.g. genes), so my issue may be a bit different: I'm talking about having two dataframes/matrices: one "A" with the data/features (the gene measurements), and another "B" with other, varied, covariates (e.g. age, sex, batch...), and the goal is to perform differential testing (not even building predictive models) in the "A" data frame, which contains the features "of interest". I could combine the "A" and "B" dataframes to perform feature selection across everything but that would be if my goal were to build a "predictive" model using some genes + the other covariates which best separate some groups; but I just want to use the genes for testing between conditions.
(although I have little experience in ML and may have misunderstood your suggestions)
I am not sure what you're trying to achieve. Since you mention linear models, you could also compute different models and select one based on an information criterion. This previous post may also be of interest.
OK, I'll start from there, thanks!