0
$\begingroup$

The Accuracy before ovesampling :

On Training : 98,54% On Testing : 98,21%

The Accuracy after ovesampling :

On Training : 77,92% On Testing : 90,44%

What does mean this and how to increase the accuracy ?

Edit:

Classes before SMOTE:

dataset['Label'].value_counts()

BENIGN           168051
Brute Force        1507
XSS                 652
Sql Injection        21

Classes after SMOTE:

BENIGN           117679 
Brute Force      117679 
XSS              117679 
Sql Injection    117679 

I used the following model:

-Random Forest : 
Train score : 0.49   Test score: 0.85
-Logistic Regression : 
Train score: 0.72    Test score: 0.93

-LSTM:
Train score: 0.79    Test score: 0.98

enter image description here

$\endgroup$
9
  • $\begingroup$ It depends on the kind of imbalance you have. It seems oversampling is not needed, but again it depends on amount of imbalance present. For example if 99,9%-0.01% then highly imbalanced and not much can be done $\endgroup$
    – Nikos M.
    Jun 20 at 18:05
  • $\begingroup$ I used SMOTE, and I used this method because some class are very low compared to some other, for example the sum of class_3 is only 21, and the sum of class_1 is 168051. $\endgroup$
    – Mimi
    Jun 20 at 19:04
  • $\begingroup$ This is weird. The accuracy on test set is highe then on the training set. What is the imbalance ratio ? How many samples in train and test set ? $\endgroup$ Jun 21 at 4:04
  • $\begingroup$ Dear @AkashDubey, The dataset split is 80% 20%. For the imbalance ratio, I did not specify this argument to SMOTE, but after Oversampling, the number of samples in each class is same. NB. SMOTE is applied only to Training set. $\endgroup$
    – Mimi
    Jun 21 at 12:10
  • $\begingroup$ @Mimi Possible argument, please look at this : stackoverflow.com/questions/51464591/… Also, this : stats.stackexchange.com/questions/59630/… $\endgroup$ Jun 21 at 12:14
0
$\begingroup$

This is weird that test accuracy is greater than the training accuracy. However, the plausible observation/explanation after looking at the distribution of classes are :

  1. The classes are highly imbalance for a multi-class classification setting.
  2. You are using SMOTE for oversampling. You can also try Adaptive Synthetic Sampling Approach for Imbalanced Learning (ADASYN) and check if the result improves.
  3. However, the most important thing is : You should try to optimise for Recall or F1 score given your classes are highly imbalanced. Since, Accuracy is not a preferred metric in a highly imbalanced classification problem. I would recommend optimising for Recall.

Possible recommendations :

  1. Hyperparameter tuning
  2. Better regularisation
  3. K-Fold cross validation.
  4. Make sure that train, validation and test sets are different.
$\endgroup$
5
  • $\begingroup$ Thank you so much, ADASYN was tried, it gives almost the same result. What did you mean by 'Better regularization' ? $\endgroup$
    – Mimi
    Jun 21 at 13:44
  • $\begingroup$ When you are using Logistic regression, what are you setting the penalty argument as ? Similarly, for random forest, are you setting the depth of the tree etc $\endgroup$ Jun 21 at 13:51
  • $\begingroup$ -RandomForestClassifier(n_estimators=200, class_weight='balanced', criterion='entropy', random_state= 0, verbose= 1, max_depth=2) -LogisticRegression(class_weight='auto') $\endgroup$
    – Mimi
    Jun 21 at 13:57
  • $\begingroup$ Choose these hyper params using a k fold cross validation on a validation set and optimise for recall. $\endgroup$ Jun 21 at 14:00
  • $\begingroup$ cv_scores = [6.73861319e-01, 2.82771953e-01, 5.32073548e-01, 7.60492017e-01, 1.38087803e-04] $\endgroup$
    – Mimi
    Jun 21 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.