99% accuracy in train and 96% in test is too much overfitting?

Question

I have a binary classification problem, the classes are quite balanced (57%-43%), with a GridSearch with Random Forest Classifier I obtained the best hyperparameters and I applied the model to train and test. Now I have 99% accuracy on train and 96% on test. Is it too much overfitting? Is it a problem?

Just for information this is my param_grid for the GridSearch

param_grid = {'n_estimators' : [100, 300, 500, 800, 1200],
'max_depth' : [5, 8, 15, 25],
'min_samples_split' : [2, 5, 10],
'min_samples_leaf' : [1, 2, 5]}

Best hyperparameters:

max_depth=25,min_samples_leaf=1,min_samples_split=2,n_estimators=1200

X_train is 102864 rows × 23 columns.
X_test is 25976 rows × 23 columns.

Answer 1

It is normal for a model to perform better on the training set than on the test set because the model has seen the training data during training and has learned to make predictions on it. However, a large difference in performance between the training set and the test set can indicate overfitting.

In your case, the difference in accuracy between the training set and the test set (99% vs 96%) is not very large, so it is highly unlikely that the model is severely overfitting. However, it is still a good idea to check for other signs of overfitting, such as a large gap between the training and validation accuracies or a decrease in performance on the test set as the model complexity increases.

Answer 2

It is likely that your model is not overfitting. I assume that the problem itself was not very difficult for the machine, and that is why you have really good results. If you had a high training accuracy and a low test accuracy that would be a sign of overfitting, however your test accuracy is high. There is a small chance that the model is overfitting. This can happen if your training data is very similar to your test data (the similarity of the data is making the solution easy, while the problem is difficult) or if you have a data leak (data from the training set is leaking into the test set). Another way to test if the model is overfitting is to use the model on new unseen data. If it performs well, it is not overfitting.

Answer 3

Based on the information that you provided, it seems that your model is achieving very high accuracy on the training set (99%) but somewhat lower accuracy on the test set (96%). This difference between the training and test accuracy could be a sign of overfitting.

To further investigate whether overfitting is occurring, you can try the following approaches:

Plot the model's performance on the training and test sets over time (e.g. using learning curves). If the model's performance on the training set is consistently much higher than its performance on the test set, it could be a sign of overfitting.

Use cross-validation to get a more accurate estimate of the model's generalization performance. You can use the cross_val_score function from scikit-learn to evaluate the model's performance using different subsets of the training data.

Try using a simpler model or applying regularization to the model. As I mentioned in my previous response, using a simpler model or applying regularization can help reduce the risk of overfitting.

Check the class balance of your data. If the classes in your data are very imbalanced (e.g. one class is much larger than the other), it can be more difficult for the model to generalize to the minority class. You can try balancing the classes in your data by oversampling or undersampling to see if it improves the model's performance on the test set.

Answer 4

A significantly higher accuracy on the training set than the test set is generally an indication of overfitting. In your case, the difference in accuracy between the train and test sets is relatively small (3%), which may suggest that your model is not severely overfitting. However, it's still important to investigate whether overfitting is a problem.

One way to investigate overfitting is to check the model's performance on a validation set (if you have one) or a separate holdout set. If the performance of the model on the validation or holdout set is significantly worse than on the training set, then this would suggest that the model is overfitting. In this case, you may need to consider using techniques to reduce overfitting, such as regularisation, early stopping, or reducing the complexity of the model.

Another approach to investigate overfitting is to look at the learning curve of the model, which shows how the model's performance changes as the size of the training set increases. If the learning curve shows that the performance on the training set is significantly better than the performance on the validation set as the size of the training set increases, then this would suggest that the model is overfitting.