This is a test version of Biostars. For the public version, visit https://www.biostars.org.
ROC Analysis for a Single Continuous Biomarker

Beginner question about ROC/AUC for a single biomarker

Hello,

I am working on a biomarker prediction problem with:

  • a derivation cohort
  • an independent validation cohort
  • a binary outcome (disease vs no disease)
  • a single continuous biomarker variable

Initially, I implemented the following workflow:

  1. In the derivation cohort, perform LOOCV logistic regression using the biomarker as the only predictor
  2. Obtain predicted probabilities for all left-out samples
  3. Compute ROC/AUC from those probabilities
  4. Train a final logistic regression model on the full derivation cohort
  5. Apply it to the validation cohort and compute validation ROC/AUC

However, I started wondering whether this is actually necessary when there is only one continuous predictor.

Since ROC curves can be computed directly from the biomarker values themselves using something like:

roc(outcome, biomarker)

would it make more sense to:

  • directly compute ROC/AUC from the raw biomarker values in the derivation cohort
  • and then independently compute ROC/AUC from the same biomarker values in the validation cohort

instead of fitting logistic regression models?

For example, if I fit:

glm(disease ~ biomarker, family = binomial)

And since ROC/AUC is rank-based, I would expect:

  • roc(disease, biomarker)
  • and roc(disease, predicted_probabilities)

to give nearly identical AUCs.

So my questions are:

  • in the single biomarker case, what exactly is the model learning that justifies LOOCV?
  • is LOOCV actually protecting against meaningful overfitting here?

Thanks!

roc auc biomarker learning machine
• 145 views • link
updated by i.sudbery i.sudbery • written by fnepo • fnepo

1 answer

in the single biomarker case, what exactly is the model learning that justifies LOOCV?

It learns where the best place to put the threshold between the two classes is, but since you are going to be examining this by ROCAUC, which doesn't use the threshold anyway, that isn't really relevant. You also learn other useful statistical properties of the dataset, but again, not neccessarily useful if you are only going to use the rank of the probabilities, rather than the probabilities themselves.

is LOOCV actually protecting against meaningful overfitting here?

This is no reason you can't overfit to a single predictor. Its just that with many predictors, the chances of overfitting are higher. But LOOCV in this case is going to be mostly useful for detecting overfitting, rather than correcting it, and since you have a validation set, its likely it would be detected there anyway.

And since ROC/AUC is rank-based, I would expect:

roc(disease, biomarker)
and roc(disease, predicted_probabilities)

to give nearly identical AUCs.

This should be very easy to test.

• 0 views • link
written by i.sudbery i.sudbery

Log in to answer this question.

More posts like this