2
$\begingroup$

I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response variable to the 100 or so variables.

My problem is that it is often difficult to compare two or more datasets and attribute the response variable to the 100 or so variables, due the large number of variables.

What do people do when faced with such a problem? What are some common analyses for doing this kind of work? It's preferable if the solution can be explained to people with a non-technical background.

$\endgroup$
0
$\begingroup$

One option is to calculate a distance metric between the data sets.

Choosing a distance metric depends on the properties of the data. If the data is binary, the Hamming distance can work. If the data are sets, the Jaccard distance can work. Other data typee require other distance metrics.

$\endgroup$
0
$\begingroup$

I would follow two avenues:

  1. Combine/concatenate all your datasets and perform some kind of dimensionality reduction (e.g. TSNE, PCA). This can help you visualise which datasets are similar to each other.
  2. Your work seems to belong to the class of attribution methods or more recently called explainable AI. This field tries to find the relative contribution of each feature/variable to the target variable. Have a look at this e-book, especially chapter 5.Book. There are methods that can calculate the variable attributions even with no access to the model prediction function (i.e. the function that maps the features to the response variable).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.