How to handle Overfitting

Question

I am working on machine learning classification problem with two classes (0/1). I would like to build a prediction model. The problem is that I have a small dataSet of shape=(89, 21) which may caused over-fitting. problem (20 independent variables). I notice that the results are highly affected by the train data and test data sizes( i.e. how the split was done ). the best results with LR was 0.90 and the worst I got 0.74.

Algo

I Split data using this instruction :

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.37, random_state=2)

then I did an upsimpling of training set since it was unbalanced {0}=62 {1}=27

the table below contain the best results on the test-set I got (test error). $$\begin{array}{c|c} & \text{Accuracy} & \text{ROC } \\ \hline \text{LogisticRegression } & 0.909091 & 0.928571 & \\ \hline \text{DecisionTree } & 0.939394 & 0.934524 & \\ \hline \text{adaBoost } & 0.848485 & 0.845238 & \\ \hline \text{RandomForest } & 0.878788 & 0.928571 & \\ \hline \text{GradientBoosting } & 0.848485 & 0.827381 & \\ \hline \end{array}$$ the next table contain the training error $$\begin{array}{c|c} & \text{Accuracy} & \text{ROC } \\ \hline \text{LogisticRegression } & 0.963415 & 0.963415 & \\ \hline \text{DecisionTree } & 1.00 & 1.00 & \\ \hline \text{adaBoost } & 1.00 & 1.00 & \\ \hline \text{RandomForest } & 1.00 & 1.00 & \\ \hline \text{GradientBoosting } & 1.00 & 1.00 & \\ \hline \end{array}$$ I need some guideline (ideas, tutorials,...) about how to manage the over-fitting problem . Thanks

Answer 1

This is a very general question, however, there are many different solutions as explained below. For your case, probably, item 2 is not the case because you can not gather a large number of data points. I would recommend using solutions 1, 3, 5, and 6 (I see you used this method but try to combine it with other solutions such as cross-validation, regularization, and feature selection).

1. Cross-validation: Use the initial training dataset to produce multiple mini train-test splits. Use these splits to tune your model.

For example, in k-fold cross-validation, partition the data into k subsets. Then, train the model on k-1 folds iteratively while using the remaining fold as the test set. In this way, you can use the cross-validation to tune hyperparameters with only the original training set.

2. Train with more data: Try to use more data points if possible.

3. Perform feature selection: There are many algorithms that you can use to perform feature selection and prevent from overfitting

4. Early stopping: When you’re training a learning algorithm iteratively, you can measure how well each iteration of the model performs.

Up until a certain number of iterations, new iterations improve the model. After that point, however, the model’s ability to generalize can weaken as it begins to overfit the training data.

5. Use regularization. As also will be discussed in item 7, the higher the complexity of the model the higher the chance of overfitting. For example, in the case of logistic regression, when the weights are large, the model gets complicated and it probably won't work on the unseen test dataset. Regularization helps to decrease the weights and so the complexity of the model.

6. Use ensembling methods such as Random Forest and Gradient Boosting. One of the main issues with decision trees is that they are prone to overfitting; i.e., high variance, it means that they work very well on training data but not on the unseen test dataset. One solution to prevent overfitting in the decision tree is to use ensembling methods such as Random Forest, which uses the majority votes for a large number of decision trees trained on different random subsets of the data.

7. Simplifying the model: very complex models are prone to overfitting. Decrease the complexity of the model to avoid overfitting. For example, in deep neural networks, the chance of overfitting is very high when the data is not large. Therefore, decreasing the complexity of the neural networks (e.g., reducing the number of hidden layers) could help to prevent overfitting.

8. Drop out method. In deep neural networks, randomly dropping some of the connections between layers by multiplying noise sampled from a Bernoulli distribution could help to prevent overfitting.

Answer 2

There are generic methods to avoid overfitting, but I'd like to address your specific problem.

Like you said, your dataset doesn't have a lot of examples compared to the number of features.
This, on its own, increases the risk of overfitting, especially if you use a more complex model such as GradientBoost or RandomForest (I'm not sure I'd use either when my number of samples is only 4x times the number of features).

So, the first thing to do would be to try and reduce the number of features.
Any model with regularization can help you with that, preferably L1 regularization and not L2. In the sklearn implementation of Logistic Regression (see the docs), you just set penalty='l1', and try you can make the regularization stronger by reducing the parameter C (or you can select C automatically by cross-validation, which I would do; see LogisticRegressionCV)

After fitting such a model (don't forget to scale your features!), you can check which features have the smallest coefficients (some would be zero, hopefully) and remove them.

This step would help any model, including those more complex than logistic regression... though, again, a simple model carries a smaller risk of overfitting than a complex one, and with your errors being what they are (if I understand your post correctly), I see no incentive to go with something more complex than logistic regression... until you get yourself more data to train a more complex model!