1
$\begingroup$

Is there a population similarity index of some kind which could help me determine if two populations in two different datasets are the same or at least similar?

The datasets have the exact same number of variables.

I would like a measure which assess the similarity variable-by-variable, but an overall measure works too.

The context of my question is that I would like to know if a variable is suitable or not for using in a classification method given the similarity of it between populations.

$\endgroup$
1
$\begingroup$

What I'm gonna say might seem too simple, but I assume, it might not be bad to fit a multivariate distribution (like Gaussian) to each distribution and then figure out what the mean and covariance matrix are. Mean might not depict so much information but the variance and correlation that are exposed in the covariance matrix might be helpful.

For the case of determining if a variable is useful or not, analyzing its correlation with other parameters might turn out to be useful.

(These are my own thoughts)

| improve this answer | |
$\endgroup$
1
$\begingroup$

Not sure if this will help you, but there is something called Gower similarity that works with different data types. It can be used to compute dissimilarity between pairs of observations in two distinct datasets on a variable-by-variable basis, which is then aggregated across all variables to create a dissimilarity index for each pair of observations from the two datasets. Don't know if anyone has used it for a problem like yours, but it seems that it should be feasible to modify the algorithm so as to aggregate the results across pairs of observations (rather than across variables) to arrive at a variable-specific value.

In R, there is a StatMatch package (https://cran.r-project.org/web/packages/StatMatch/StatMatch.pdf) that is able to handle this multiple dataset scenario. There is information in the documentation on how it computes distances between observations for different data types and shows how the aggregation works. The aggregation is simply a weighted sum of dissimilarities for two observations across variables, so the idea would be to change it to a weighted sum of dissimilarities for the same variable across all pairs of observations.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.