I am currently using caret which optimises for accuracy. Is it possible to optimise for sensitivity. I see documentation mentioning that metric = 'roc' has been used, but sensitivity does not appear to be one of the options.
Probably. It depends on what you are doing exactly, and in particular which summaryFunction you are using. If it's twoClassSummary for common binary classification then you should be able to specify
metric='Sensitivity'. If that's not the case you also have the option of overriding the default performance summaryFunction with your own implementation that calculates the metrics you desire.
I highly recommend reading through the documentation at http://topepo.github.io/caret/training.html#control
As long as you have:
summaryFunction = twoClassSummary
trainControl object that you have set, then you can set:
metric = "Sens" # or "ROC" or "Spec" whichever you desire
when you are fitting the model:
control <- trainControl(method = "repeatedcv", number = 5, repeats = 5, summaryFunction = twoClassSummary, classProbs = T, savePredictions = "final") logreg.fit <- train(Class ~., data = training_data, method = "glm", family = "binomial", trControl = control, metric = "Sens")
Max mentions in his book that tuning models for sensitivity may reveal improvements in this metric by tuning models towards its optimisation, i.e. you may be able to fit a model with improved sensitivity that has very minor, if any, trade offs in specificity. This is in the particular case of class imbalance in the data.