High probabilities of success for wrong predictions

Question

I'm studying the behavior of machine failures in a production scenario. For this, I generated random data to form my imbalanced training set, consisting of categorical data, which indicate whether or not there was a failure in each subperiod. The failures were generated according to a exponential distribution. I have 24 features (Period_1 to Period_24), each containing information about the historical failures for 448 subperiods. Furthermore, I have three more features consisting of Temperature, Moisture, and Pressure (generated using the Normal distribution). My intention is to predict the behavior of the failures for the next period based on these features.

I used the ROC metric and considered several strategies to deal with unbalanced data, such as oversampling, undersampling, ROSE, and ADASYN. Furthermore, I tried to use ensemble to improve performance. I tested all of the following models: gradient boosting algorithm, random forest, Classification and Regression Trees, neural networks, Bagged CART, SVM, C5.0, eXtreme Gradient Boosting, and k-Nearest Neighbors. I also tried to use regularized models but none of these strategies worked. The best result obtained was using the model "SVMRadial" considering resampling with the ROSE package. In this case, ROC = 0.7614, Sensitivity = 0.7639, and Specificity = 0.6065 for the training set and Sensitivity = 0.75, and Specificity = 0.6914 for the test set (the latter obtained through the Confusion Matrix). However, when making predictions, the trained model is resulting in high probabilities for wrong predictions. So, I would like to know if this is a problem of the training model. Also, would anyone have any idea how to improve these results?

Any help will be appreciated.

A sample of the data:

The code for the chosen model:

set.seed(123);
    
partition <- createDataPartition(data_failures$Period_24, p = 0.8, list = F)
trainingSet <- data_failures[partition,]
testingSet <- data_failures[-partition,]
    
train.control <- trainControl(method = "repeatedcv", number = 10, repeats = 3, sampling = "rose", classProbs = TRUE, summaryFunction = twoClassSummary)
    
model_24 <- train(Period_24 ~., data = trainingSet, method = "svmRadial", preProc = "zv", metric = "ROC", trControl = train.control)
    
print(model_24)
    
predictions <- predict(model_24, newdata = testingSet)
print(confusionMatrix(predictions, testingSet$Period_24))

I'm using R considering the caret package.

Answer 1

It is difficult to diagnose without having more info. But based upon what you mention I point:

1. Since you have a class imbalance, you MUST use regularizarided models that penalize the cost function made by the minority class, more strictly than those errors for the majority class (Boosting models, Linear models)

2. Try using another metric like ROC AUC or Balanced Accuracy to measure the performance of your model so that you can see an adequate point of view of how your model is performing. An additional advantage of using ROC Score is that you can find the "optimal" threshold being the one that maximizes Area under the roc curve when you make predictions with your model

3. This is the most important part. Use a technique to debug your model like SHAP values,Partial dependence plot in order to understand how your model is learning and the relationships between the features and the target

4. If you are using ensemble methods, you can set something named Monotonicity constraints That will allow you to add business knowledge to the model in order to correct potential non-sense relationships that the model may find