Clustering a set of matrices with kmeans/other approaches

Question

I am posting here as overflow community told me it was better to ask it here. I was trying to cluster a set of matrices in order to obtain $n$ clusters based on matrix similarity.

In practice, my dataset is composed by matrices $r\times c$ with discrete values $\{-1, 0, 1\}$. The scope is obtain $n$ clusters which group similar matrices. For example, setting $r = 4, c = 3$, my data looks like:

[[[0, 0, 1],
 [0, -1, 0],
 [0, 0, 0],
 [0, 1, 0]],

 [[0, 0, -1],
 [0, 0, 0],
 [0, -1, 0],
 [0, 1, 1]],

 [[1, 0, 1],
 [0, -1, 0],
 [1, -1, 0],
 [0, 1, 0]],

 ...]

I've used a kmeans clustering approach, but unfortunately it does not seems to work well. I'll introduce some code to ease my explanation:

def get_kmeans_clusters(heatmaps, n_cluster): # DOWNSIDE: k is not known, data is treated without ordering!
    # Get from heatmap dictionary all values
    data = list(heatmaps.values())

    # Flatten matrices in order to ease kmeans
    data = np.array([mat.flatten() for mat in data])
    
    # Cluster
    kmeans = KMeans(n_clusters=n_cluster, random_state=0).fit(data.reshape(-1,1))
    
    ...

Simply, I'm giving the matrices as flatten elements to kmeans, without any preprocessing whatsoever.

But I incur in an error when trying to apply kmeans:

Number of distinct clusters (2) found smaller than n_clusters (4). Possibly due to duplicate points in X.

This image will further clarify my current naive approach:

The image represent one of my matrix (it is a matrix representation of a set of Reinforcement Learning policies actions during a day column wise, for being extra-clear). I take the matrices as they are and flatten them in the data array. In my mind, the matrices are already the set of features I want to use to describe each single sample in my dataset.

My naive assumption is that my matrices are flattened and kmeans is not preserving the ordering of the elements (i.e. for kmeans [0,1,1,-1,0] == [1,1,0,-1,0] clustering wise). Moreover, I believe that the number of features is just overwhelming with respect to the number of data samples (513 samples with 6000 features each).

For this reason, I wanted to approach the problem in a different way, or at least I would like to understand if I am doing something wrong. I've tried to do some data exploration, and I have found that each matrix (heatmap) in my dataset is unique (i.e. $\nexists x_i, x_j \in X: x_i = x_j$, where $X$ is the set of heatmaps). I was wondering if there is some way to describe my data in a different way.

For example, I wanted to keep track of image geometry (where the elements $1, -1$ are placed) by using a CNN and taking the hidden layers as a compact representation of my heatmaps, but I do not really know if this could be any helpful.

From here, my question: is there any reliable clustering approach for multidimensional data like matrices? Is kmeans really meaningful in this context?

Answer 1

This looks more like a categorical variables and if you use default k means with euclidean it may not give you very good results.

For categorical data, the Euclidean distance measure used by K-Means reduces to counting the number of variables on which two cases disagree. After the initial centers are chosen (which depends on the order of the cases), the centers are still binary data. For the first iteration, as the cases are compared to cluster means, they will always be at some integer distance from each of the centers. There will often be ties, and the case will be assigned to a cluster in an arbitrary manner. Using Euclidean distance (the only measure available to K-Means), it is impossible to overcome the symmetry and break the ties in any meaningful way.

You may try following :

1. Using Cosine Similarity instead of euclidean distance

2. Using algorithm which provides distance measures for only categorical data