Overfitting in the trained model

Question

For my project on the classification problem of predicting churn customers, I trained various base models using k-fold validation on the training dataset and out of which random forest gave the best result but during prediction it is giving lower score than it gives on the test data which is apparently showing overfitting, even SVC was showing the same thing. How can I correct my code so that it gives similar results to that of the training data set?

Python code:

models = []

models.append(('LR', LogisticRegression()))

models.append(('KNN', KNeighborsClassifier(n_neighbors=5))) 

models.append(('TREES', DecisionTreeClassifier(max_depth=20)))

models.append(('FOREST', RandomForestClassifier(max_depth=15, n_estimators=30)))

models.append(('SVM', SVC()))

models

names = []

for name, model in models:

    kfold = StratifiedKFold(n_splits=5, random_state=1, shuffle=True)

    cv_results = cross_val_score(model, X_train_scaled, Y_train, cv=kfold, scoring='roc_auc')

    print(cv_results)

    mean_score = np.mean(cv_results)

    print(f"Average ROC_AUC Score for {name} is: {mean_score:.3f}")

output : [0.8627102 0.84906579 0.84063598 0.82388159 0.83202681]
Average ROC_AUC Score for FOREST is: 0.842*

rf_model=RandomForestClassifier(max_depth=15,n_estimators=30)

rf_model.fit(X_train_scaled,Y_train)

y_pred=rf_model.predict(X_test_scaled)

auc_score= roc_auc_score(Y_test,y_pred)

print(f'AUC Score for test dataset is: {auc_score}')

Output: AUC Score for test dataset is: 0.7205451341221849

Answer 1

It seems like your model is overfitting on the training data, which means it's not generalizing well to new, unseen data. And there could be three reasons for that:

1. Your train and test dataset is sampled from completely different data sources. Please check that.
2. Your model is overfitting on the training data because it is learning the noise and patterns in the training data too well and resulting in poor generalization to new, unseen data.
3. The model could overfit when the model is too complex or when there is not enough training data.

You might try the following techniques:

1. Feature selection: You can do some feature selection on your dataset. Use RFE or train a random forest model and then plot the feature importance and select only those features which are contributing towards the model score.

2. Hyperparameter tuning: Perform a grid search or random search to find the best hyperparameters for your Random Forest model.

3. Regularization: For the SVC model, you can try increasing the regularization parameter (C) to reduce overfitting. A higher value of C will create a wider margin, which may result in better generalization to the test data.

4. Ensemble methods: You can try combining multiple models to create a more robust classifier. You can also try bagging, boosting, or stacking to combine the predictions of multiple base models.

5. Increase the size of your training dataset: If possible, collect more data to train your model. A larger dataset can help in reducing overfitting and improving the model's performance on the test data.

6. Feature engineering: Create new features from the existing ones, which might help in improving the model's performance. For example, you can create interaction features, polynomial features, or apply transformations like log, square root, etc.

Hope it will help.