I've been trying to model a dataset using various classifiers. The response is highly imbalanced (binary) and I have both numerical and categorical variables, so I applied SMOTENC and Random oversampling methods on Training set. In addition, I used a Validation set to tune the models parameters by GridSearchCV(). As both precision and recall were important for me, I used f1 to find the best model.
I should note that I selected these three subsets by cluster analysis and extracting samples by stratified train_test_split() from each cluster; so I have more confident that the subsets have more similarity.
Due to complex nature of Decision Tree and Random Forest or boosting techniques, I usually get high fitting (high f1 score) on Training set, relatively high on Validation set, but moderate to low on Test set.
The general sign for overfitting is the high difference between Training and Test sets (or between Validation and Test sets in my problem); but I am confused how to select the best model in following cases:
Case A: Training fit is very high; but Validation and Test sets fit are low but close to each other
Case B: Training, Validation, and Test fits are similar; but much lower than Case A.
F1 Score Model Train Val Test ---------------------------------------- A: SVC 80.1 60.3 37.5 B: MLPClassifier: 43.2 40.0 39.1
I know that Case A might be the best model,however there is no guarantee that is produces similar result for new data, but which model do you pick with regards to overfitting? (assume that precision and recalls for both models are similar)