I am currently using caret which optimises for accuracy. Is it possible to optimise for sensitivity. I see documentation mentioning that metric = 'roc' has been used, but sensitivity does not appear to be one of the options.


Probably. It depends on what you are doing exactly, and in particular which summaryFunction you are using. If it's twoClassSummary for common binary classification then you should be able to specify metric='Sensitivity'. If that's not the case you also have the option of overriding the default performance summaryFunction with your own implementation that calculates the metrics you desire.

I highly recommend reading through the documentation at http://topepo.github.io/caret/training.html#control


As long as you have:

summaryFunction = twoClassSummary

in the trainControl object that you have set, then you can set:

metric = "Sens"  # or "ROC" or "Spec" whichever you desire

when you are fitting the model:

control <- trainControl(method = "repeatedcv", number = 5, repeats = 5, 
                        summaryFunction = twoClassSummary, classProbs = T, 
                        savePredictions = "final")

logreg.fit <- train(Class ~., data = training_data, method = "glm",
                    family = "binomial", trControl = control, metric = "Sens")

Max mentions in his book that tuning models for sensitivity may reveal improvements in this metric by tuning models towards its optimisation, i.e. you may be able to fit a model with improved sensitivity that has very minor, if any, trade offs in specificity. This is in the particular case of class imbalance in the data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.