0
$\begingroup$

I'm studying the behavior of machine failures in a production scenario. For this, I generated random data to form my imbalanced training set, consisting of categorical data, which indicate whether or not there was a failure in each subperiod. The failures were generated according to a exponential distribution. I have 24 features (Period_1 to Period_24), each containing information about the historical failures for 448 subperiods. Furthermore, I have three more features consisting of Temperature, Moisture, and Pressure (generated using the Normal distribution). My intention is to predict the behavior of the failures for the next period based on these features.

I used the ROC metric and considered several strategies to deal with unbalanced data, such as oversampling, undersampling, ROSE, and ADASYN. Furthermore, I tried to use ensemble to improve performance. I tested all of the following models: gradient boosting algorithm, random forest, Classification and Regression Trees, neural networks, Bagged CART, SVM, C5.0, eXtreme Gradient Boosting, and k-Nearest Neighbors. I also tried to use regularized models but none of these strategies worked. The best result obtained was using the model "SVMRadial" considering resampling with the ROSE package. In this case, ROC = 0.7614, Sensitivity = 0.7639, and Specificity = 0.6065 for the training set and Sensitivity = 0.75, and Specificity = 0.6914 for the test set (the latter obtained through the Confusion Matrix). However, when making predictions, the trained model is resulting in high probabilities for wrong predictions. So, I would like to know if this is a problem of the training model. Also, would anyone have any idea how to improve these results?

Any help will be appreciated.

A sample of the data: enter image description here

The code for the chosen model:

set.seed(123);
    
partition <- createDataPartition(data_failures$Period_24, p = 0.8, list = F)
trainingSet <- data_failures[partition,]
testingSet <- data_failures[-partition,]
    
train.control <- trainControl(method = "repeatedcv", number = 10, repeats = 3, sampling = "rose", classProbs = TRUE, summaryFunction = twoClassSummary)
    
model_24 <- train(Period_24 ~., data = trainingSet, method = "svmRadial", preProc = "zv", metric = "ROC", trControl = train.control)
    
print(model_24)
    
predictions <- predict(model_24, newdata = testingSet)
print(confusionMatrix(predictions, testingSet$Period_24))

I'm using R considering the caret package.

$\endgroup$
1
  • $\begingroup$ Try Using Platt scaling (en.wikipedia.org/wiki/Platt_scaling) this may help to align the predicted probabilities with the prior (sample) probabilities. This must be used after model has been trained, and also you have to use cv to make the calibration. "Platt scaling has been shown to be effective for SVMs as well as other types of classification models, including boosted models and even naive Bayes classifiers, which produce distorted probability distributions" $\endgroup$ – Julio Jesus Aug 17 '20 at 19:08
1
$\begingroup$

It is difficult to diagnose without having more info. But based upon what you mention I point:

  1. Since you have a class imbalance, you MUST use regularizarided models that penalize the cost function made by the minority class, more strictly than those errors for the majority class (Boosting models, Linear models)

  2. Try using another metric like ROC AUC or Balanced Accuracy to measure the performance of your model so that you can see an adequate point of view of how your model is performing. An additional advantage of using ROC Score is that you can find the "optimal" threshold being the one that maximizes Area under the roc curve when you make predictions with your model

  3. This is the most important part. Use a technique to debug your model like SHAP values,Partial dependence plot in order to understand how your model is learning and the relationships between the features and the target

  4. If you are using ensemble methods, you can set something named Monotonicity constraints That will allow you to add business knowledge to the model in order to correct potential non-sense relationships that the model may find

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.