The Accuracy before ovesampling :

On Training : 98,54% On Testing : 98,21%

The Accuracy after ovesampling :

On Training : 77,92% On Testing : 90,44%

What does mean this and how to increase the accuracy ?


Classes before SMOTE:


BENIGN           168051
Brute Force        1507
XSS                 652
Sql Injection        21

Classes after SMOTE:

BENIGN           117679 
Brute Force      117679 
XSS              117679 
Sql Injection    117679 

I used the following model:

-Random Forest : 
Train score : 0.49   Test score: 0.85
-Logistic Regression : 
Train score: 0.72    Test score: 0.93

Train score: 0.79    Test score: 0.98

enter image description here

  • $\begingroup$ It depends on the kind of imbalance you have. It seems oversampling is not needed, but again it depends on amount of imbalance present. For example if 99,9%-0.01% then highly imbalanced and not much can be done $\endgroup$
    – Nikos M.
    Jun 20, 2021 at 18:05
  • $\begingroup$ I used SMOTE, and I used this method because some class are very low compared to some other, for example the sum of class_3 is only 21, and the sum of class_1 is 168051. $\endgroup$
    – Mimi
    Jun 20, 2021 at 19:04
  • $\begingroup$ This is weird. The accuracy on test set is highe then on the training set. What is the imbalance ratio ? How many samples in train and test set ? $\endgroup$ Jun 21, 2021 at 4:04
  • $\begingroup$ Dear @AkashDubey, The dataset split is 80% 20%. For the imbalance ratio, I did not specify this argument to SMOTE, but after Oversampling, the number of samples in each class is same. NB. SMOTE is applied only to Training set. $\endgroup$
    – Mimi
    Jun 21, 2021 at 12:10
  • 1
    $\begingroup$ Why would you want to oversample instead of modeling probabilities? $\endgroup$
    – Dave
    Jun 21, 2021 at 13:55

2 Answers 2


Accuracy is not a very good metric generally, but especially in the presence of serious class imbalance. In your case, always predicting BENIGN would achieve an accuracy of 98.72%, but is useless, while your models might be useful despite having lower accuracy.

That oversampling hurts your training accuracy is natural. The largest effect of oversampling is that the predicted probabilities are shifted, and if you are predicting the class as the one with largest predicted probability, this will result in many more predictions of the minority classes, which is wrong from an accuracy point of view. (One thing wasn't made clear in the post: are you measuring performance on the resampled data, or the original? The resampled data won't suffer from the effect above, but might well suffer from poor predictions of the tiny Sql injection class, which might not have enough signal to properly identify.)

As Dave says in a comment, best to start without oversampling, and create a metric that properly captures the cost/benefit tradeoff of true/false positives/negatives for each class. Then it might be beneficial to oversample, but if you are using the predicted probabilities, that is unlikely to give huge lift.


This is weird that test accuracy is greater than the training accuracy. However, the plausible observation/explanation after looking at the distribution of classes are :

  1. The classes are highly imbalance for a multi-class classification setting.
  2. You are using SMOTE for oversampling. You can also try Adaptive Synthetic Sampling Approach for Imbalanced Learning (ADASYN) and check if the result improves.
  3. However, the most important thing is : You should try to optimise for Recall or F1 score given your classes are highly imbalanced. Since, Accuracy is not a preferred metric in a highly imbalanced classification problem. I would recommend optimising for Recall.

Possible recommendations :

  1. Hyperparameter tuning
  2. Better regularisation
  3. K-Fold cross validation.
  4. Make sure that train, validation and test sets are different.
  • $\begingroup$ Thank you so much, ADASYN was tried, it gives almost the same result. What did you mean by 'Better regularization' ? $\endgroup$
    – Mimi
    Jun 21, 2021 at 13:44
  • $\begingroup$ When you are using Logistic regression, what are you setting the penalty argument as ? Similarly, for random forest, are you setting the depth of the tree etc $\endgroup$ Jun 21, 2021 at 13:51
  • $\begingroup$ -RandomForestClassifier(n_estimators=200, class_weight='balanced', criterion='entropy', random_state= 0, verbose= 1, max_depth=2) -LogisticRegression(class_weight='auto') $\endgroup$
    – Mimi
    Jun 21, 2021 at 13:57
  • $\begingroup$ Choose these hyper params using a k fold cross validation on a validation set and optimise for recall. $\endgroup$ Jun 21, 2021 at 14:00
  • $\begingroup$ cv_scores = [6.73861319e-01, 2.82771953e-01, 5.32073548e-01, 7.60492017e-01, 1.38087803e-04] $\endgroup$
    – Mimi
    Jun 21, 2021 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.