In my problem, step one is to build a model to classify cases as one of True or False (1 or 0 could also be used obviously). Once the optimum model is found, step two is to retrieve probabilities for these predictions and use these to work out the optimum threshold (based on an accuracy measure) to classify my data thereafter.
So I know for step two I need to use a classification accuracy measure e.g. f1 score to work out the optimum threshold however, I am debating on the best approach for step one. My initial thought was to do a two-class classification model. The model I'm using is a Radial SVM implemented in R. My issue with this however, is using
caret, how I get predicted scores once the optimum model is found with
classProbs=T (full code below). I don't know if this is ideal as I read somewhere (can't find link now sorry!), to do this,
caret actually runs a second model which doesn't seem like a great idea to me. The only information on this is the package documentation seems to be
classProbs a logical; should class probabilities be computed for classification models (along with predicted values) in each resample?
I have also searched SO for another way to get predicted values using caret but have had no success.
My other idea was to use a regression model for step one using, 0 and 1 as the only y values and then perhaps implementing a custom error function that allows for predicted values outside of 0 and 1. My thought here is that I don't necessarily want to penalise a model if the prediction is above 1 and the actual is 1.
I'm not sure which approach is better or if there is a third way I'm not thinking of, any help/suggestions would be great! I think the first approach is probably better if the class probabilities calculation is not done through using a separate model so if anyone has found further documentation on what goes on with this setting that would also be great!
svm_tests <- train(x = x_train, y = y_train, method = "svmRadial", scale=F, tuneGrid = expand.grid("C"=0.1,"sigma"= 0.05), trControl = trainControl("repeatedcv", #Do 5 fold cv, repeated 5 times with different seeds repeats = 5, number = 5, seeds = seeds, summaryFunction = m.c.c, classProbs=T, #Used to get probabilities when using predict )) scores_svm <- predict(svm_tests, x_valid, type = "prob")