Assume that we have two small sets of feature vectors (each feature vector representing an item). What is a good way of finding which features have the biggest difference (in distribution) between the two (small, 50 samples) sets? Given that the distribution of the data per feature is not necessarily normal.
A reformulation of your question would be:
Which are the features that differentiate the two datasets more accurately, because they differ a lot across them?
A common way to answer this is by checking the information gain of the attributes, based on the reduction of the entropy of the dataset. The attributes with the highest information gain are the ones that separate the dataset more accurately.
Assuming you have m attributes, you need to:
- Label your samples based on which dataset they belong (A or B)
- Attempt to split the dataset based on the values of each attribute. This means you need to perform m splits.
- The split that maximizes the information gain, thus minimizes the individual sub-entropy of the two splitted groups, shows the attribute that separates the data the best. This attribute is the one that differs the most in both datasets and is the answer to your question.
FYI this is exactly what is done when training decision trees, to cut out (prune) features that are not informative enough in order to:
- Reduce the tree's variance
- Reduce training / inferencing time