Hi all,
To identify remote homology, I have used different tools using benchmark dataset for which I have the labels for the true homologs. I have calculated the AUC values based on True positives and False positives in the homology result for each classifier.
The AUC values for different classifiers are like this
Classifier 1 -0.601
Classifier 2 - 0.732
Classifier 3 - 0.752
Classifier 4 - 0.828
Classifier 5 - 0.883
What is the best way to assign weights to each classifier (as they have different performance) so that I can multiply these weights to the normalized scores of each individual tools to find the aggregated score?
Thanks in advance
What is the best way to assign weights to each classifier
It depends on what quantity you are trying to optimize. If you are interested in maximum AUC value in the end, the weights should be based on that quantity. Maximum AUC doesn't necessarily mean maximum accuracy, for example. Nor does it mean minimum log-loss value.
AUC is a ranking-based measure that in general works the best for imbalanced datasets. The threshold for determining true/false homologs may not be at probability=0.5, but it could be at p=0.8 instead.
Thanks for your answer. I am interested in giving weights to each tool that I have used for detecting the homologs for my query proteins as stated in "https://www.nature.com/articles/srep32333#Sec6".
I am unable to find that how are they assigning weights to each tool.
Log in to answer this question.
