Comparing two data distributions, not the probability distribution

Question

Let's assume that I have the 200 MNIST data. These 200 data are divided into two training dataset. one training dataset has 100 and another training dataset has the remaining 100.

I trained my two CNN models with the first training dataset and second training dataset. After training, the testing error rates were different from each other since the model has a different training dataset. So, I want to identify this difference by looking for the two data distributions. I came up with my solution to identify the difference between the two data distributions.

1. Summing all data from each label (For example, each label has 10 data respectively and summing the 10 data from each label, thus 10 outputs should be calculated.)
2. Divide by the number of labels
3. Calculate cosine similarity and summing all the similarity values.

So, this algorithm performs two times, one is for the first training dataset and another one is for the second training dataset and lastly, I should check the difference between the two similarity values. Actually, I can understand by looking at similarity difference the reason why the testing error rates are different from each other, but sometimes I see big different testing error rates even though the similarity of the two dataset are close to each other.

Let me know if you have some better idea or a well-known method. I searched similar method such as Kullback Leibler Divergence and Siamese Networks, but I think these two methods are not proper to solve this problem.

Thank you

Answer 1

The similarity between datasets cannot explain the test error of neural networks, the optimization can go very differently for slightly different datasets