0
$\begingroup$

I am all new with ML. I try to understand what is Kfold and cross_val_score.
I made this model:

RandomForestRegressor(max_depth=17,n_estimators=93,criterion='mse')
modelfinal.fit(xtrain1, ytrain1)
mrse_test = np.sqrt(mean_squared_error(y_pred=modelfinal.predict(xtest1), y_true=ytest1))

It gave me:

Train/test R2 mae mrse
Train 0.94 13.52 22.07
Test 0.605 34.57 56.592

Because on the train, it gave me 0.9 as R2 and only 0.6 on test. I have been told I should use Kfold cv to see if I have over overfitting or not. But I just can't understand how it will help me, and what I did is right or not. This is what I did:

cv=ShuffleSplit(5,test_size=0.2)
score=cross_val_score(modelfinal, X, Y, cv=cv)

This is what I get:

array([0.59618812, 0.57799759, 0.55214982, 0.63904499, 0.65027832])

What can I conclude from this?

$\endgroup$

2 Answers 2

1
$\begingroup$

Overfitting is visible when the performance on the training set is much higher than the test set, so you clearly had overfitting with your first experiment already.

Cross-validation is not a way to see overfitting by itself, it's a way to obtain a more reliable performance value by using several splits (removes chance factor) and using an overall larger test set (the full data).

One thing I notice in your experiment is that the max_depth parameter is set to 17 which is very high, especially in case your training set is not too large. This means that you're allowing the model to create very big and complex trees, so there is more chance that it would represent small patterns which happen by chance in the data. Try reducing this parameter value, hopefully this will avoid or at least decrease the overfitting. If it's not enough there might be some work to do at the level of features: too many features and/or not enough instances can cause overfitting as well.

$\endgroup$
3
  • $\begingroup$ Thank you very much for your answer, before choosing 17 i tried to change the parameter max_depth and i choose 17 cause it gave me 0.9 0.6 i thought it is better than 0.8 0.6 or 0.7 0.5! $\endgroup$
    – Rita
    Commented Jan 9, 2021 at 12:14
  • $\begingroup$ @Rita It's true that it's better to have high performance of course, but on the test set. the pairs of values that you mention all show overfitting, so probably the source of overfitting is not (or not only) the max depth parameter. you should probably simplify or reduce the features. $\endgroup$
    – Erwan
    Commented Jan 9, 2021 at 12:55
  • $\begingroup$ thank you for your help i'll work on it $\endgroup$
    – Rita
    Commented Jan 9, 2021 at 16:29
1
$\begingroup$

Kfold will not help you to understand whether your model is getting overfitted or not.

Currently the reason was that you have used max_depth to be as 17 and hence it just highly overfitted on training_set and provided bad performance on the test set.

What You can try is with current model lower that max_depth value and check it changes on the test set. If you are okay to change the model , then try using LGBMRegressor or XGBRegressor , they have eval_set as a parameter , so they examine the increase in performance on testing data while training , and when it starts to worsen it stops and provide you with the model that provides best performance on that test set with a given set of parameters.

I hope this helps.

$\endgroup$
1
  • $\begingroup$ thank you so much for your answer , i'll try to change the model and hope it will work $\endgroup$
    – Rita
    Commented Jan 9, 2021 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.