I working on a binary classification task. The dataset is quite small ~1800 rows and ~60 columns. There are no duplicates in the rows. I am comparing different classifiers amongst the canonical ones: random forest, logistic regression, boosted tree and SVC. I am training the hyperparameters by a CV on 90% (train) with 10% held out to measure the generalization error (test). The dataset is slightly unbalances (1 to 3 ratio of classes) hence I used a stratified fold for all splits. I also use roc-auc as a metric for my CV.
I get the following results for roc-auc score and accuracy:
DummyClassifier Train ROC-AUC score: 0.50000 Accuracy: 0.69705 Test ROC-AUC score: 0.50000 Accuracy: 0.69545 LogisticRegression Train ROC-AUC score: 0.88459 Accuracy: 0.78666 Test ROC-AUC score: 0.72559 Accuracy: 0.69545 RandomForestClassifier Train ROC-AUC score: 1.00000 Accuracy: 0.99695 Test ROC-AUC score: 0.81748 Accuracy: 0.80455 XGBClassifier Train ROC-AUC score: 1.00000 Accuracy: 0.99949 Test ROC-AUC score: 0.80617 Accuracy: 0.79545 SVC Train ROC-AUC score: 0.89900 Accuracy: 0.83248 Test ROC-AUC score: 0.73515 Accuracy: 0.73182
There is always a significant gap between train and test scores. I am clearly overfitting. I guess it is a consequence of the low number of rows but I am not sure about what to do about that? Force the CV grid search for hyperparameters to a range with strong regularization?