1
$\begingroup$

I have two models, A and B, trained on Imagenet. Their accuracies on Imagenet validation set are 35.6% and 28.64% respectively, while the accuracy of their ensemble (averaging their scores) is 35.68%. I am interested in finding out why the ensembling isn't effective here.

Specifically, I was going to inspect the confusion matrices for each model, but Imagenet has more than 1000 classes which makes this intractable. Another thing that was suggested to me is Mutual Information, but I can't figure out how to apply it in this context.

So, I have a two part question:

  1. Why doesn't the accuracy of the ensemble degrade (to the average of two accuracies) or improve?
  2. Is there a way of visualizing/scoring the output of the two networks to measure correlation?

Edit 1: Both are AlexNet models, but were trained with two different pre-trained weight initializations. The pre-trained weights themselves come from two different self-supervised tasks. Also, when these models were trained (initialized with respective pre-trained weights) on Pascal, there's is a significant boost in the accuracy. Thus, my quest to figure out how do you measure the correlation between models that are ensembling.

$\endgroup$
  • 1
    $\begingroup$ It did improve from 35.6 to 35.68 ? $\endgroup$ – Isbister Feb 7 '19 at 16:20
  • $\begingroup$ Yes, it did. But, it seems too small an improvement 0.08%. $\endgroup$ – Ajinkya Feb 7 '19 at 16:31
  • $\begingroup$ So it is working then. It's not causing any harm. There is no guaranty at all that there would be a larger improvement. $\endgroup$ – Isbister Feb 7 '19 at 16:33
  • $\begingroup$ Yes, I understand that, but I am more interested in why the accuracy only improved marginally. Had the accuracy improved significantly I know what's happening. Same for reduction in accuracy. I am looking for an alternative to confusion matrix. Something that lets me see what's happening. $\endgroup$ – Ajinkya Feb 7 '19 at 16:37
  • 1
    $\begingroup$ How identical are the models? Do they have the same features, same structure? If they have learnt different things then there could be a higher increase because they complement each other. Otherwise it might just be information overlap. Have a look at their shap values github.com/slundberg/shap ? $\endgroup$ – Isbister Feb 7 '19 at 17:20
2
$\begingroup$

From what you are saying on can make up that you predict scores for each class and that the highest score is your class prediction.

A question you may ask is whether or not the class prediction is affected by the ensembling at all. For the sake of the example consider an example where Model A predicts a class always with a normalised score of 1.0 (i.e. full confidence, recommended reading). Suppose Model A is right 80% of the time. Model B however, is less certain. It places his bets on the top 5 classes that it deems likely with an equal normalised score of 0.2. Averaging the scores results in the same top predicted class as for Model A alone. You could easily check this by calculating the correlations between the predictions of the models with the ensembled predictions.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Looking at each model's accuracy is a good place to start, of course, but you should also look at the relationship between their predictions.

I'd start with something simple, like checking only the prediction correctness (similar to a confusion matrix) - i.e. both correct / only A correct / only B correct / both incorrect.
This would allow you to immediately see if, for example, model B only gets it right when model A does (or more generally, to see if the models are actually quite dependent). If you have many classes, it'll be a bit hard to do the same but by class.

Another thing to look at is to see where the averaged model is different than model A; if both model A and the averaged model are very similar in accuracy, there'll be only a few examples when their predictions are different - and you can inspect those predictions.

And another thought, that might not apply to your case but might apply to other readers - when looking at accuracy, it's good to have an idea of your class sizes. I'm guessing that for your dataset you don't have 35% of your examples in one class, but it could be that one model just outputs a very high score for the largest class, and averaging it with another model does not change the result.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.