If I have 10,000 training samples then what should I do:

Bootstrapping and train 10 classifiers on it and then aggregating


randomly divide the dataset into 10 parts and train 10 classifiers on them and then aggregating. Which will be better?

Will the 2nd method reduce variance and will it be better than 1st method


I think the second method will yield less correlated models than the first method. It is particularly true with decision trees which tend to quickly overfitting in the bottom nodes. It will help reduce the variance.

However, by using the second approach, you will end with 10 smaller datasets and so you risk to introduce a variance error due to the too small number of observations. Discussing about decision trees again, it would mean that your trees algorithms will tend to overfitting upper in the tree. And so you will increase your variance error.

In my opinion, for most of datasets, it is still better to use the first approach than the second. I think that very low correlated estimators won't bring a better improvement that the first method.

We can also observe that differences in the two approaches depend also of the number of observations, the number of features, the kind of estimators you are using.. A benchmark would be really interesting !


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.