3
$\begingroup$

I'm trying to understand what I did wrong when trying to answer this question. The exact question is:

Assume that we have 3 trained prediction models, and each model outputs either -1 or 1. We then tested the accuracies of these models and obtained the following outcomes:

Model Accuracy
m1 0.60
m2 0.55
m3 0.45

Let M be the ensemble model that outputs a plurality vote of these three models. If we assume that the errors of the models m1, m2, and m3 are independent, what is the probability that M(x) would be correct on a test instance x?

I thought that because this was a plurality vote, and the classification errors are independent of each other, I could simply take the weighted average of the accuracy of the three classifiers:

$ \begin{align*} P(X) &= \sum_{all\ models\ M_j} P(C_i|x,M_j)P(M_j) \\ &= \frac{1}{L} \sum_{all\ models\ M_j} P(C_i|x,M_j) \\ &= \frac{1}{3}(0.60+0.55+0.45)\\ &= 0.53 \end{align*} $

But I was told that this is incorrect (with no context as to why).

Can someone explain why this is incorrect? If this is a plurality vote (which to me assumes that the votes of each classifier are equal), why can I not simply take the weighted average?

$\endgroup$
1
  • $\begingroup$ Wait, how can a binary classifier be 0.45 accurate? That would mean taking the opposite answer is 0.55 accurate. No wonder the ensemble is less accurate than the best model in it. $\endgroup$ – towr Apr 19 at 11:29
5
$\begingroup$

It's not the actual data, it's the probabilities. So you should consider all the scenarios of voting.

For the Ensemble to be correct,
Either any two or all the three should be correct

=$[m_1*m_2*(1- m_3) + m_1*(1-m_2)*m_3) + (1-m_1)*m_2*m_3] + [m_1*m_2*m_3]$

= [0.6*0.55*(1-0.45) + 0.6*(1-0.55)*0.45 + (1-0.6)*0.55*0.45] + 0.6*0.55*0.45

= 0.5505

$\endgroup$
0
2
$\begingroup$

You would get a correct prediction if model's 1, 2 and 3 were

  • Correct, Correct, Correct

  • Correct, Correct, Wrong

  • Correct, Wrong, Correct

  • Wrong, Correct, Correct

Likewise, You would get a wrong prediction if they were

  • Wrong, Wrong, Correct
  • Wrong, Correct, Wrong
  • Correct, Wrong, Wrong
  • Wrong, Wrong, Wrong

So, a total of 8 possibilities.

The likelihood of the first event C, C, C is 0.6 × 0.55 × 0.45

The likelihood of the fifth event W, W, C is 0.4 × 0.45 × 0.45

You want to calculate likelihood of events 1-4 and 5-8 and sum them up separately. You also want to total them all 1-8. The first sum divided the total gives you the answer you are looking for.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.