1
$\begingroup$

I've been doing some research on ensemble learning and read that for base models, model with high variance are often recommended (can't remember which book I read this from exactly).

But, it seems counter-intuitive because wouldn't having base models with low variance(doing good on test set) be better than having multiple bad base models?

$\endgroup$
0
$\begingroup$

Intuitively speaking, ensembles benefit most from diversity.

Imagine being in a room of people making a decision together. If everyone more or less agrees, you don't benefit from having more people at the table. But if people tend to have different opinions, when they DO agree, it is a stronger message that the decision must be correct.

The same applies to ensembles. Models with high variance are more likely to produce different predictions, which will improve the quality of the prediction. High variance also minimizes the risk that multiple models are all wrong at the same time, based on the assumption that models are right more often than wrong.

$\endgroup$
  • $\begingroup$ Yes, I was thinking the way as you. However having multiple different opinions mean having multiple models(possible good predictors) that have different opinions. So having variant models is good, I understand that however if each model has high variance, it would be room full of different opinions that are highly incorrect is it not? Therefore my conclusion was, each base model should be different from each other however individual base model should have low variance. $\endgroup$ – Ambleu Jan 12 at 8:02
  • 1
    $\begingroup$ The key here is that at equal performance, it is better to have base models with high variance than high bias. Mathematically speaking, as long as models perform better than random, good predictions will end up outweighing bad predictions (when using more and more base models) $\endgroup$ – Valentin Calomme Jan 12 at 8:15
  • $\begingroup$ Ah I see, so you should accurate base models however if there are two accurate models where one has high bias and one has high variance than you should choose the latter? $\endgroup$ – Ambleu Jan 12 at 8:34
  • 1
    $\begingroup$ Precisely! Higher variance base models will generally benefit the ensemble more than high bias ones due to the added "diversity" $\endgroup$ – Valentin Calomme Jan 12 at 8:51
  • 1
    $\begingroup$ Thanks for great explanation! $\endgroup$ – Ambleu Jan 12 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.