1
$\begingroup$

I've been doing some research on ensemble learning and read that for base models, model with high variance are often recommended (can't remember which book I read this from exactly).

But, it seems counter-intuitive because wouldn't having base models with low variance(doing good on test set) be better than having multiple bad base models?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Intuitively speaking, ensembles benefit most from diversity.

Imagine being in a room of people making a decision together. If everyone more or less agrees, you don't benefit from having more people at the table. But if people tend to have different opinions, when they DO agree, it is a stronger message that the decision must be correct.

The same applies to ensembles. Models with high variance are more likely to produce different predictions, which will improve the quality of the prediction. High variance also minimizes the risk that multiple models are all wrong at the same time, based on the assumption that models are right more often than wrong.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Yes, I was thinking the way as you. However having multiple different opinions mean having multiple models(possible good predictors) that have different opinions. So having variant models is good, I understand that however if each model has high variance, it would be room full of different opinions that are highly incorrect is it not? Therefore my conclusion was, each base model should be different from each other however individual base model should have low variance. $\endgroup$
    – haneulkim
    Commented Jan 12, 2021 at 8:02
  • 2
    $\begingroup$ The key here is that at equal performance, it is better to have base models with high variance than high bias. Mathematically speaking, as long as models perform better than random, good predictions will end up outweighing bad predictions (when using more and more base models) $\endgroup$
    – Valentin Calomme
    Commented Jan 12, 2021 at 8:15
  • $\begingroup$ Ah I see, so you should accurate base models however if there are two accurate models where one has high bias and one has high variance than you should choose the latter? $\endgroup$
    – haneulkim
    Commented Jan 12, 2021 at 8:34
  • 1
    $\begingroup$ Precisely! Higher variance base models will generally benefit the ensemble more than high bias ones due to the added "diversity" $\endgroup$
    – Valentin Calomme
    Commented Jan 12, 2021 at 8:51
  • 1
    $\begingroup$ Thanks for great explanation! $\endgroup$
    – haneulkim
    Commented Jan 12, 2021 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.