0
$\begingroup$

I have a table with 100K+ rows and 100+ columns all numeric. Rather than using k-means to cluster rows together (and creating a new column category that labels each row), I want to cluster the columns/variables together. Is there a Python clustering library or example that I can use to set k and cluster variables?

$\endgroup$
3
  • $\begingroup$ Simple 2 steps: First, transform the data-table (dataframe) in python; This will make columns as rows and rows as columns. And use the new data-table to do row-clustering using k-means. pandas.pydata.org/pandas-docs/stable/reference/api/… $\endgroup$
    – DataFramed
    Jul 15, 2021 at 14:12
  • 1
    $\begingroup$ @DataFramed with 100k+ rows which will become 100k+ features I'd expect your approach to strongly suffer from the curse of dimensionality. $\endgroup$
    – Jonathan
    Jul 15, 2021 at 15:08
  • $\begingroup$ @Sammy @bitSandwich21, we need to use dimensionality reduction techniques on the new dataframe to overcome the curse of dimensionality e.g. PCA techniques. Agree, the dataset shape after transformation will suffer from the curse of dimensionality. $\endgroup$
    – DataFramed
    Jul 16, 2021 at 3:37

1 Answer 1

0
$\begingroup$

There is an implementation in Scikit-learn named FeatureAgglomeration, that does exactly what you want but using Agglomerative clustering

It simply runs the cluster algorithm in the transposed matrix of X.

So in your case you could apply this idea but using Kmeans instead

Update:

I recently came across a similar problem for dimensionality reduction and I found a Python implementation of an originally SAS procedure named VarClus.

According to the package's documentation:

This is a Python module to perform variable clustering (varclus) with a hierarchical structure. Varclus is a nice dimension reduction algorithm. Here is a short description:

  1. A cluster is chosen for splitting.
  2. The chosen cluster is split into two clusters by finding the first two principal components, performing an orthoblique rotation, and assigning each variable to the rotated component with which it has the higher squared correlation.
  3. Variables are iteratively reassigned to clusters to maximize the variance accounted for by the cluster components.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.