Hi,
I need insight on how to improve my predictive models and validation. In this study, I build predictive models to determine best predictor gene/genes that could predict treatment response which are responder or non-responder ( R or NR). I have been doing this for months and following these links Building a predictive model by a list of genes, How to exclude some of breast cancer subtypes just by looking at gene expression? and Resources for gene signature creation to build predictive models on my discovery cohort (n= 55 responder , NR = 32 R=23).
Initially, I performed DESeq2 to find statistically significant DEGs then I narrowed down the DEGs using logistic, lasso and elastic regression and keep the best 10 genes to be included in the final predictive model. ROC for discovery cohort is >90%.
However when I validated the final models on my independent cohort (n= 98, NR= 22, R= 76), the ROC is between 50-60% for all models.
I also have tried random forest, where I used RFE for feature selections on significant DEGs, however the result remain the same.
Please advise me what next to do to improve my predictive models and validation results.
Best wishes
1 answer
I narrowed down the DEGs using logistic, lasso and elastic regression and keep the best 10 genes to be included in the final predictive model. ROC for discovery cohort is >90%.
I don't know what narrowed down the DEGs using logistic, lasso and elastic regression means, but I assume you refer to feature selection. If so, the best 10 genes doesn't strike me as enough to discriminate for the future samples. I'd start from 100 best genes and maybe decrease that number in steps of 10, and see where that gets you.
ROC AUC is not expressed in percent values, but rather as a fraction on a [0,1] scale.
When you say that ROC is >90%, is that based on cross-validation? If not, how exactly did you determine this value?
However when I validated the final models on my independent cohort (n= 98, NR= 22, R= 76), the ROC is between 50-60% for all models.
It is notable that you had a balanced training dataset, while the unseen data is less so. That will likely contribute to a somewhat lower score, but should not make a large difference that you see. Generally speaking, training datasets are larger than validation datasets, while in your case you have roughly a 1:2 ratio. If everything else fails, you may have to repeat the training on 100-ish samples, and verify on the 50-ish. That way you will boost your training cohort and likely improve the predictive ability.
I also have tried random forest, where I used RFE for feature selections on significant DEGs, however the result remain the same.
If you used top 10 genes again, I would expect the same result.
Log in to answer this question.
