I have a table with 100K+ rows and 100+ columns all numeric. Rather than using k-means to cluster rows together (and creating a new column category that labels each row), I want to cluster the columns/variables together. Is there a Python clustering library or example that I can use to set k and cluster variables?
$\begingroup$
$\endgroup$
3
1 Answer
$\begingroup$
$\endgroup$
There is an implementation in Scikit-learn named FeatureAgglomeration, that does exactly what you want but using Agglomerative clustering
It simply runs the cluster algorithm in the transposed matrix of X.
So in your case you could apply this idea but using Kmeans instead
Update:
I recently came across a similar problem for dimensionality reduction and I found a Python implementation of an originally SAS procedure named VarClus.
According to the package's documentation:
This is a Python module to perform variable clustering (varclus) with a hierarchical structure. Varclus is a nice dimension reduction algorithm. Here is a short description:
- A cluster is chosen for splitting.
- The chosen cluster is split into two clusters by finding the first two principal components, performing an orthoblique rotation, and assigning each variable to the rotated component with which it has the higher squared correlation.
- Variables are iteratively reassigned to clusters to maximize the variance accounted for by the cluster components.
k-means. pandas.pydata.org/pandas-docs/stable/reference/api/… $\endgroup$use dimensionality reduction techniqueson the new dataframe to overcome thecurse of dimensionalitye.g.PCA techniques. Agree, the dataset shape after transformation will suffer from the curse of dimensionality. $\endgroup$