4
$\begingroup$

When using ensembling methods for regression, a common approach is to average (using the arithmetic mean) the outputs of the weak learners in order to obtain the output of the ensemble. Is there a reason why the average that people take is the arithmetic mean? Why not performing a geometric or harmonic mean?

$\endgroup$
2
$\begingroup$

We can answer this overarching question by exploring a couple sub-questions:

What are the properties of popular averaging formulae?

Geometric: $ \space $ $ \space $ $ \begin{equation} \bigg(\displaystyle \prod^N_{i=1} x_{i}\bigg)^{\frac{1}{N}} \end{equation} $

We can see that any set that contains a zero has a geometric mean of zero.

Also, processing negative numbers is awkward as they can become imaginary. Even if negative numbers do not result in an imaginary mean (e.g. the set has an odd number of elements), the result is often not what we expect an "average" to be. For instance, given a set, let's take some averages:

$ \{-1,2,4\} $

Arithmetic mean: $ 1.\overline{6} $

Geometric mean: $ -2 $

The geometric mean is clearly not what we would expect for an "average". This is because the geometric mean is primarily used for understanding the multiplicative differences in a set, which is useful for averaging data on differing scales.

Let's say we only consider the absolute value of the elements. Then, given a set below, we still get undesirable results:

$ \{2^0,2^1,...2^{10}\} $

Arithmetic mean: $ 186.\overline{09} $

Geometric mean: $ 32 $

From this example we can see that where the arithmetic mean is linear, the geometric mean is logarithmic.


Harmonic: $ \space $ $ \space $ $ \begin{equation} N \cdot \bigg({\displaystyle \sum^N_{i=1} x^{-1}_{i}}\bigg)^{-1} \end{equation} $

Like the geometric mean, the harmonic mean of a set containing zero is uninformative. In fact, it is undefined.

A more interesting property of the harmonic mean is that it discriminates against outliers. For instance, given a set, let's again take some averages:

$ \{ 1e^{-5}, 1e^{-5}, 1e^{10}\}$

Arithmetic mean: $ 3.\overline{3}e^9 $

Harmonic mean: $ 1.5e^{-5} $


An easier way of thinking about the averages is like this:

Where arithmetic is linear, geometric is logarithmic, and harmonic is reciprocal.


Should every learner have an equal representation in the final vote?

If you are using the arithmetic mean, the weak learners are treated equally. This is generally desired as you do not know which learner is closer to the correct answer. Using the geometric or harmonic mean would imply that you want to "weight" the set of weak learner outputs according to their respective additive structures.

As a side-note, you can (and probably should) use machine learning algorithms, such as a MLP or SVM, to combine the outputs of your ensemble, which would probably yield superior results to simple averaging.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I'd caution against using more complicated ensemble methods to model your data before trying simple models, however. There are many real-world settings where simple ensembles will perform just as well as (or better than) more complex models. I'd argue that simple average models should always be used as baselines against which more complex models should be compared. $\endgroup$ – Nicholas G Reich Oct 2 '18 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.