1
$\begingroup$

I am building a classification model based on some machine performance data. Unfortunately for me, it seems to over-fit no matter what I change. The dataset is quite large so I'll share the final feature importance & cross validation scores after feature selection.

#preparing the data    
X = df.drop('target', axis='columns')
y = df['target']

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.75, random_state=10, stratify=y)

enter image description here

I then cross validate as follows;

logreg=LogisticRegression()
kf=KFold(n_splits=25)
score=cross_val_score(logreg,X,y,cv=kf)

print("Cross Validation Scores: {}".format(score))
print("Average Cross Validation score : {}".format(score.mean()))

Here are the results that I get:

>  Cross Validation Scores [1.  1.  1.  1.  1.  1.   1.  1.  1.  1.  1.  1.  1.  1. 1.  1.  1.  1.  0.94814175 1.  1.  1.   1.  1.  1.] 
> Average Cross Validation score : 0.9979256698357821

When I run RandomForests, the accuracy is 100%. What could be the problem? PS. The classes were imbalanced so I "randomly" under-sampled the majority class.

UPDATE: I overcame this challenge by eliminating some features from the final dataset. I retrained my models using a few features at a time and was able to find out the ones that caused the "over-fitting". In short, better feature selection did the trick.

$\endgroup$

1 Answer 1

2
$\begingroup$

This isn't overfitting. You're reporting cross-validation scores as very high (and are not reporting training set scores, which are presumably also very high); your model is just performing very well (on unseen data).

That said, you should be asking yourself if something is wrong. There are two common culprits that come to mind:

  1. One of your features is very informative, but wouldn't be available at prediction time ("future information", or in the extreme case, you accidentally left the target variable in the independent variable dataframe)
  2. Your train-test splits don't respect some grouping (in the extreme case, rows of the frame are repeated and show up in both training and test folds).

Otherwise, it's entirely possible your problem is just easily solved by your model.

See also
Why does my model produce too good to be true output?
Quote on too good to be true model performance?

$\endgroup$
2
  • $\begingroup$ Thanks there, Ben. 1. I checked my train-test split and clearly separated the test & training sets; but ended up with similar results. I also checked to ensure that I didn't leave the target in the test set. 2. Do you think it'd be wise for me (help the model to generalize better) to reduce the labels further by removing the 4 features with the highest importance? 3. However, this gets me thinking - could it be that the data is too similar for these models to pick on a pattern? $\endgroup$ Oct 25, 2022 at 10:25
  • $\begingroup$ @KholoNuwagabaCollins from what I can tell from your description, this is just a good model (/easy problem); there is nothing to fix. $\endgroup$
    – Ben Reiniger
    Oct 25, 2022 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.