Im new to StackExchange, so i am sorry if this is not the right way to ask a question on StackExhange.
For my thesis I wish to propose a methode for future research on using PCA to cluster features (feature clustering) and then apply per-cluster PCA. I got the idea from this paper: this paper. But I have a hard time finding literature about PCA being used to cluster variables (not reduce variables). I could imagine that it is not ideal to use PCA to cluster variables but i still would like to propose the method. Do any of you know any literature, articles, books etc.
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Not quite clear on what do you expect PCA to do. You have features A, B, C, D, E. Lets assume that some of them are not linearly independent. Lets assume that only A, C, E are independent. Your PCA will suggest that 3 out of 5 dimensions is enough to capture variance. Now what? $\endgroup$– CryoCommented May 9, 2022 at 21:11
-
$\begingroup$ I think it's a good idea for a thesis. But if you extend to factor analysis, that is already thought of as clustering in the marketing world. $\endgroup$– Ralph WintersCommented May 9, 2022 at 23:36
-
$\begingroup$ Hi @Cryo, thanks for your response! Basicly i have two steps in my proposed method, 1. cluster variables that are (Strongly) correlated. 2. Apply pca on each cluster independently. Basicly i am looking for literature on the first step, how would one cluster variables using PCA. Let's use your example of features A till E. Features A and B have a (strong) correlation, C and D would also have a (strong) correlation and feature E has a weak correlation with any other feature. PC1 (AB explains most of the variance), PC2(CD), PC3(E). This could mean 3 clusters. $\endgroup$– aryanCommented May 10, 2022 at 7:37
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
I am not sure PCA is quite what you are after. I think it may help to visualize what you are after. I think the image is as follows for 5 features with 2 records (i.e. 2 rows 5 columns):
Here, because there are only two records, features are 2D vectors. Would you be after capturing A, B, C in one cluster and D, E in another?
If so, I think you should simply do eigenvalue decomposition of the covariance matrix. That will give you the eigenvectors along which you have greatest variance. For the example above, I would have 5x5 covariance matrix, with two eigenvectors that have large eigenvalues, and 3 more that have very small eigenvalues.
The eigenvectors with large eigenvalues are then your clustering target, project your feature vectors along these eigenvectors. e.g. if $V_1$ and $V_2$ are your two eigenvectors, then compute magnitudes of dot-products $\left|A.V_1\right|$ and $\left|A.V_2\right|$. Then assign feature $A$ to cluster 1 if $\left|A.V_1\right|>\left|A.V_2\right|$ and vice versa. Depending on your data it may be a good idea to group features that are aligned with eigenvectors that have very similar eigenvalues (as a form of regularization).
PS: PCA does similar things but is geared toward a different purpose (i.e. some of the information provided by SVD of a design matrix, is similar to what you get from diagonalization of the covariance matrix)
$\begingroup$
$\endgroup$
Looks like you want go for unsupervised methods for feature selection. You can use PCA but it might not be that effective.
I would suggest to go through these links.
https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/
https://scikit-learn.org/stable/modules/feature_selection.html
There is a method called Principal Feature Analysis in Link 4. You can have a look at that. If you are using R, there is sparcl package for sparse clustering.
