# Understanding why ensembling only improves marginally

I have two models, A and B, trained on Imagenet. Their accuracies on Imagenet validation set are 35.6% and 28.64% respectively, while the accuracy of their ensemble (averaging their scores) is 35.68%. I am interested in finding out why the ensembling isn't effective here.

Specifically, I was going to inspect the confusion matrices for each model, but Imagenet has more than 1000 classes which makes this intractable. Another thing that was suggested to me is Mutual Information, but I can't figure out how to apply it in this context.

So, I have a two part question:

1. Why doesn't the accuracy of the ensemble degrade (to the average of two accuracies) or improve?
2. Is there a way of visualizing/scoring the output of the two networks to measure correlation?

Edit 1: Both are AlexNet models, but were trained with two different pre-trained weight initializations. The pre-trained weights themselves come from two different self-supervised tasks. Also, when these models were trained (initialized with respective pre-trained weights) on Pascal, there's is a significant boost in the accuracy. Thus, my quest to figure out how do you measure the correlation between models that are ensembling.

• It did improve from 35.6 to 35.68 ? Commented Feb 7, 2019 at 16:20
• Yes, it did. But, it seems too small an improvement 0.08%. Commented Feb 7, 2019 at 16:31
• So it is working then. It's not causing any harm. There is no guaranty at all that there would be a larger improvement. Commented Feb 7, 2019 at 16:33
• Yes, I understand that, but I am more interested in why the accuracy only improved marginally. Had the accuracy improved significantly I know what's happening. Same for reduction in accuracy. I am looking for an alternative to confusion matrix. Something that lets me see what's happening. Commented Feb 7, 2019 at 16:37
• How identical are the models? Do they have the same features, same structure? If they have learnt different things then there could be a higher increase because they complement each other. Otherwise it might just be information overlap. Have a look at their shap values github.com/slundberg/shap ? Commented Feb 7, 2019 at 17:20