High probabilities of success for wrong predictions

I'm studying the behavior of machine failures in a production scenario. For this, I generated random data to form my imbalanced training set, consisting of categorical data, which indicate whether or not there was a failure in each subperiod. The failures were generated according to a exponential distribution. I have 24 features (Period_1 to Period_24), each containing information about the historical failures for 448 subperiods. Furthermore, I have three more features consisting of Temperature, Moisture, and Pressure (generated using the Normal distribution). My intention is to predict the behavior of the failures for the next period based on these features.

I used the ROC metric and considered several strategies to deal with unbalanced data, such as oversampling, undersampling, ROSE, and ADASYN. Furthermore, I tried to use ensemble to improve performance. I tested all of the following models: gradient boosting algorithm, random forest, Classification and Regression Trees, neural networks, Bagged CART, SVM, C5.0, eXtreme Gradient Boosting, and k-Nearest Neighbors. I also tried to use regularized models but none of these strategies worked. The best result obtained was using the model "SVMRadial" considering resampling with the ROSE package. In this case, ROC = 0.7614, Sensitivity = 0.7639, and Specificity = 0.6065 for the training set and Sensitivity = 0.75, and Specificity = 0.6914 for the test set (the latter obtained through the Confusion Matrix). However, when making predictions, the trained model is resulting in high probabilities for wrong predictions. So, I would like to know if this is a problem of the training model. Also, would anyone have any idea how to improve these results?

Any help will be appreciated.

A sample of the data:

The code for the chosen model:

set.seed(123);

partition <- createDataPartition(data_failures$Period_24, p = 0.8, list = F) trainingSet <- data_failures[partition,] testingSet <- data_failures[-partition,] train.control <- trainControl(method = "repeatedcv", number = 10, repeats = 3, sampling = "rose", classProbs = TRUE, summaryFunction = twoClassSummary) model_24 <- train(Period_24 ~., data = trainingSet, method = "svmRadial", preProc = "zv", metric = "ROC", trControl = train.control) print(model_24) predictions <- predict(model_24, newdata = testingSet) print(confusionMatrix(predictions, testingSet$Period_24))


I'm using R considering the caret package.

• Try Using Platt scaling (en.wikipedia.org/wiki/Platt_scaling) this may help to align the predicted probabilities with the prior (sample) probabilities. This must be used after model has been trained, and also you have to use cv to make the calibration. "Platt scaling has been shown to be effective for SVMs as well as other types of classification models, including boosted models and even naive Bayes classifiers, which produce distorted probability distributions" – Julio Jesus Aug 17 '20 at 19:08