1
$\begingroup$

So I have this dataset with about 750 variables (columns) and 50,000 rows of entries. I would like to reduce the dimensionality of the dataset to say 25-50-100 dimensions and then compute a correlation matrix between these dimensions. I have continuous and discrete columns. I would also like to be able to know which dimensions represent which bundle of single variables in the original dataset. I should add that I have no output variable, I am just trying to find correlation in the data. Is there a way to do this in Python?

$\endgroup$
0
$\begingroup$

Yes, in scikit-learn, you can find the correlation between the elements using LedoitWolf Estimator. For dimensionality reduction, I assume you will use PCA but then, you want to backtrack the reduced to the original data, for that I don't have a solution, since, PCA transforms your data and computes the largest variance direction. But, yes both of these won't require any labels.

$\endgroup$
1
  • $\begingroup$ Wow, I did not know about Ledoit-Wolf! I used factor analysis since I have some discrete and continuous data. $\endgroup$
    – Masterloop
    Feb 25 '21 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.