This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Dealing with highly correlated features for disease classification

Hi everyone,

I am working on a pathway-based classification problem. For each pathway/biological term, I calculate a score representing its activity in each sample, and I then use these pathway activity scores as features to predict disease status.

The main issue I am facing is that several of these features are highly correlated (in some cases, pairwise correlation > 0.9) (which is not particularly surprising given the biological overlap between pathways).

I initially tried logistic regression, but the high multicollinearity leads to unstable coefficients and very high VIF values. I have also tried regularized approaches such as Elastic Net, and tried Random Forest as well, but so far they do not seem to improve classification performance compared with simpler models using just one of the features.

I am therefore wondering what would be a good strategy for dealing with highly correlated pathway-level features in this setting.

Any suggestions or references to similar analyses would be greatly appreciated.

Thanks!

pathway machine prediction learning

1 answer

I'd try to engineer a composite score of these scores if their pathways are meaningfully distinct for your case. This will collapse highly correlated features into representative ones, simplifying the feature space + reducing storage, memory, and redundancy. Another approach is applying a dimensionality reduction algorithm, e.g. PCA, to extract the relevant principal components and then use those as representative features for the same outcome.

The choice of the approach depends on what you want to do down the line with these scores and classifier/regressor, e.g. is the goal to later use the model to predict certain pathway/term scores? If that's the case, then you'd need a way to go back from the composite score to a space of individual pathway/term scores, thus the composite score derivation must be bijective and you're better off engineering the composite than using PCs . If that's not a concern, e.g. you can cluster these pathways/biological terms into a superset and simply speak to the contribution of this larger cluster, then the derivation can be just injective, and dimensionality reduction is sufficient.

You can also try L1/L2 regularization before training variance-based models if you want to keep the features distinct without omission. That being said, a Random Forest model tends to be insensitive to linear correlations. If you're hitting a ceiling with it or other tree models, then I'd also look into the types of features you're including/excluding in your training+validation sets as well as hyperparameters.

References:

Chan, J. Y.-L., Leow, S. M. H., Bea, K. T., Cheng, W. K., Phoong, S. W., Hong, Z.-W., & Chen, Y.-L. (2022). Mitigating the Multicollinearity Problem and Its Machine Learning Approach: A Review. Mathematics, 10(8), 1283. https://doi.org/10.3390/math10081283

Regression Analysis: An Intuitive Guide for Using and Interpreting Linear Models - by Frost

An Introduction to Statistical Learning with Applications in R - by James, Witten, Hastie, Tibshirani

Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow by Géron

ETA: The Pivot algorithm for correlation clustering although you could use hierarchical clustering instead if these are GO terms.

Log in to answer this question.