1
$\begingroup$

I working on a binary classification task. The dataset is quite small ~1800 rows and ~60 columns. There are no duplicates in the rows. I am comparing different classifiers amongst the canonical ones: random forest, logistic regression, boosted tree and SVC. I am training the hyperparameters by a CV on 90% (train) with 10% held out to measure the generalization error (test). The dataset is slightly unbalances (1 to 3 ratio of classes) hence I used a stratified fold for all splits. I also use roc-auc as a metric for my CV.

I get the following results for roc-auc score and accuracy:

 DummyClassifier
Train
ROC-AUC score: 0.50000
Accuracy: 0.69705
Test
ROC-AUC score: 0.50000
Accuracy: 0.69545

 LogisticRegression
Train
ROC-AUC score: 0.88459
Accuracy: 0.78666
Test
ROC-AUC score: 0.72559
Accuracy: 0.69545

 RandomForestClassifier
Train
ROC-AUC score: 1.00000
Accuracy: 0.99695
Test
ROC-AUC score: 0.81748
Accuracy: 0.80455

 XGBClassifier
Train
ROC-AUC score: 1.00000
Accuracy: 0.99949
Test
ROC-AUC score: 0.80617
Accuracy: 0.79545

 SVC
Train
ROC-AUC score: 0.89900
Accuracy: 0.83248
Test
ROC-AUC score: 0.73515
Accuracy: 0.73182

There is always a significant gap between train and test scores. I am clearly overfitting. I guess it is a consequence of the low number of rows but I am not sure about what to do about that? Force the CV grid search for hyperparameters to a range with strong regularization?

$\endgroup$
  • $\begingroup$ This sort of behavior can be a result of data leakage. Check out my answer on this similar question. $\endgroup$ – tuomastik Apr 6 '18 at 2:44
  • $\begingroup$ Regarding the imbalance in the data, have you tried down-sampling/upsampling? $\endgroup$ – zacdav Apr 6 '18 at 4:52
2
$\begingroup$

For the problem of overfitting, you could look train models that employ regularization. For instance this examples shows how to regularize an SVM.

Another thing I noted is that you have used the tag "unbalanced-classes". If that is the case, accuracy isn't a very good metric. While AUC is good at this, I've personally had trouble with this metric in the past. My suggestion would be to include a metirc like F1-score and most importantly in each case calculate the confusion matrix. This will show you if you are missing one class more than the other. If that is the case you might want to incorporate an oversampling method (e.g. SMOTE) into your pipeline.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I have toyed with regularization in my CV grid search. But I fail to understand why the gap is so large between train and test score. Could it be that I have two few samples (rows) and that I am stuck in the part of the learning process that just memorizes the training data? $\endgroup$ – Learning is a mess Apr 5 '18 at 22:57
  • 3
    $\begingroup$ Yes, that is an issue but usually you can deal with this through heavy regularization. Another thing that typically has the same effect with regularization is model averaging. All these techniques will reduce your training accuracy, in hopes of improving your test accuracy $\endgroup$ – Djib2011 Apr 6 '18 at 12:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.