1
$\begingroup$

I have trained a Random forest regressor which is giving me a rmse score of 70.72. But when I tried the same model in cross_val_score with a cv of 10, it gave me an array that looks something like this.

array([63.96728974, 60.43972474, 63.98455253, 61.69770344, 94.24656396,
       59.93552448, 60.77507132, 54.20247545, 59.20367786, 61.59208032])
# min
54.20247545

This shows that the model has model potential to perform even better if the data get splitted in a certain way.

So, My question is that is there any way to find the best split which can achieve the optimal loss which we saw in cross-validation.

Sometimes I use for loop to find out the best value for random_state for best split, but this is not memory efficient and sometimes also does not work. So it will be great if there is another alternative for this!!

$\endgroup$
2
$\begingroup$

It would be a very bad idea to select the "optimal split".

The goal of cross-validation is to evaluate the model more accurately by minimizing the effect of chance due to the splitting. Selecting the "optimal split" goes against the idea of reliably estimating the performance, in fact this would purposefully overestimate the model.

It's important to realize that the goal of evaluating a model is not maximizing performance, it's to reliably estimate the performance of the model on any random test set.

$\endgroup$
3
  • $\begingroup$ But if we choose the optimal split, isn't that mean that model will learn more things. I mean I used to think that a different split allows the model to learn the different properties of the data and if the data is splited well (optimal split), and its validation accuracy is good then it learned more properties of the data, which is a step toward the generalized model. $\endgroup$ Sep 15 at 19:31
  • $\begingroup$ That's one of the reason I am looking for the way to get optimal split $\endgroup$ Sep 15 at 19:31
  • 2
    $\begingroup$ @DarkstarDream your interpretation is wrong I'm afraid: normally if the model is not overfit it should learn almost the same information from any split. However the test set is different, and the split which gives maximum performance only means that this particular test set is easier to predict than the others. So the model is (almost) the same but the performance is higher only by chance, thus it's not a good estimate of the real performance. $\endgroup$
    – Erwan
    Sep 16 at 9:58
3
$\begingroup$

I fully agree with @Erwan that this is a very bad idea.

All performance estimates based on a finite sample of data will have some amount of variation caused by sampling from an underlying population, but this is just "noise".

In terms of test-training splits, if one test training split gives you better results than another, it probably means that by chance most of the "easy" to classify examples happened to end up in the test set and most of the difficult to classify ones end up in the training set. So it will give an optimistic estimate of the model's true performance. If you collect a new test set (or observe the classifier in operational use), over time the proportion of easy and difficult to classify patterns will be different to those in the "best" partition and give a lower (less biased) accuracy.

Here is a simple thought-experiment to demonstrate that this will produce an optimistically biased performance estimate. Consider a completely random learning task, where we are trying to predict the result of flipping a fair coin (the target variable) and we have (say) ten random input attributes, that are also generated by flipping a fair coin. Now in this case, the classifier you construct on the training partition can have no possible skill in predicting the targets in the test partition, because they are completely random and have no causal relationship to any of the inputs. Say our dataset has ten patterns where the target is a head and ten where the target is a tail. We then randomly partition the data, say 100 times and train and test a classifier each time, each containing half of the data. Now the error rate on the test partition will be on average 0.5 (as it is predicting a random coin flip), but the error rate won't be 0.5 in every partition - in some cases, by random chance it will be higher than 0.5 and in some it will be lower. It is a random process, so in some cases the output of the random classifier learned on the training partition will be better correlated with the test labels than it will be in other partitions. However this variation is just random noise, so if you choose the partition with the highest accuracy, it will suggest the classifier has some actual skill in predicting a random coin flip, but we know that cannot possibly be the case. The apparent skill is spurious, and is caused by using a biased performance estimate.

I used a similar example (with code) on the stats SE here to explain why we can easily get a biased performance estimate if we perform cross-validation based feature selection and then try to use that cross-validation as an estimate of performance (it will be highly biased). It may help to read that one first.

Essentially, we shouldn't make any choices about the model or it's evaluation after looking at the test data if we want to have an unbiased performance estimate. I wrote a paper about this some time ago, because it is a very common error with model selection (e.g. optimising hyper-parameters):

Gavin C. Cawley and Nicola L. C. Talbot, "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation", Journal of Machine Learning Research, 11(70):2079−2107, 2010 (www)

$\endgroup$
1
  • 1
    $\begingroup$ great answer, your example really helped me understand it in more depth $\endgroup$ Sep 16 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.