0
$\begingroup$

I have developed a logistic regression model for a classification problem and obtained an AUC (Area Under the Curve) score of approximately 0.9. The model was estimated by splitting the available data using the common 70-30 rule (70% training data, 30% testing data).

Now, I would like to understand if my model is indeed overfitting the training data. In particular, I'm interested in understanding whether or not the good achieved model performance (e.g., ROC-AUC score ~0.9) is due to overfitting.

Could someon guide me on how to evaluate the presence of overfitting in my logistic regression model given these circumstances, please? I thought about applying my model on the testing data to obtain model-implied estimates but such approach would not lead anywhere since the real-world observations of my target variable are not available yet and won't be in the near future.

Any suggestions is welcome. Thanks in advance!

$\endgroup$
1
  • $\begingroup$ Do you have the original train set, or at least a train set similar to the original? $\endgroup$
    – Memristor
    Jun 14, 2023 at 8:08

1 Answer 1

1
$\begingroup$

Just cross-validate your model. Basically, you keep apart a portion of the training data (now called the validation data or split) which are then used for validation (same as testing but on such validation split).

There are various cross-validation methods, like the leave-one-out or the k-fold cross-validation.

In practice you can use the cross_validate function from scikit-learn:

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_validate

# this would perform (stratified) 5-fold cross-validation on ROC-AUC metric
scores = cross_validate(estimator=LogisticRegression(...), X=x_train, y=y_train,
                        cv=5, scoring='roc_auc', 
                        return_train_score=True, return_estimator=True)

And then compare the scores['train_score'] with the scores['test_score'], if the gap is large the model is likely overfitting: you get a score for each cross-val fold, so you can plot and/or aggregate them for example.

If you haven't used scikit-learn to train your logistic regression model you can still set-up the validation procedure manually.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.