# Clustering on imbalanced data that has high correlation

I am clustering images of two categories, but for the purposes of the experiment, I do not know the labels i.e. this is an unsupervised problem. Via correlation heatmaps and other experiments, I am confident that my images are highly correlated, at least via a Pearson correlation coefficient. However, I face very large imbalanced datasets in my problem, and as a result, my hierarchical agglomerative clustering fails when the imbalance passes a ratio of around $$10:1$$.

I have no idea of alternative solutions. I have tried reducing dimensionality via PCA, but this does not help.

If PCA doesn’t help, then I don’t think that your problem has to do with the correlation between the images. I think your problem is just that your classifier has trouble learning the problem properly if the classes are too imbalanced. If that’s it, then the possible solutions are:

• Get more data (always :-) )
• Undersample the large class (use only 10% of the examples of that class, for instance)
• Oversample the small class (duplicate examples from that class)
• Adjust your loss function to assign higher cost to errors on the small class

When the problem of the imbalance is addressed, I would expect PCA to allow the model to learn more easily, so I would try to put that back in.

• Thanks for answering! Questions - for the sake of the experiments, I am intentionally limited to having a cap on the size of my sample population, so getting more data would be hard. I am interested in the under and oversampling - but I am, for the purpose of the experiment, not allowed to know which samples belong to the smaller or higher class, and can thus not adjust their sizes. Is there an unsupervised approach for over and undersampling? Regarding the loss function - what exactly is analagous to the loss function in clustering, besides the distance metric? Apologies if this is obvious. – imageimbalanceuser Jun 8 '19 at 19:01

When you have high correlation problems you should go for dimensionality reduction. Multicollinear features can be "summarized" and controlled for by techniques such as PCA or Autoencoders for dimensionality reduction. I don't recomment PCA, since it can only extract factors that linearly associated with your data. In my opinion:

Autoencoders  >  PCA


There are many other dimensionality reduction techniques, but these are the most common.

Once you have reduced the dimensionality of your data you can run your cluster analysis on the reduced dataset, and observe how different observations (and their categories) will be distributed with respect to each other.

Hope this helps, otherwise let me know.

• Oh I forgot: if you can run a Neural Network on an imbalanced dataset, you can feed mini batches of training data built in a way so that different classes are more evenly distributed. In this way, the model will be "tricked" and attribute more even weights to different classes. That is if you are working with Neural Networks, of course. That doesn't apply to all ML problems. – Leevo Jun 8 '19 at 10:59
• I don’t see how t-SNE is a useful technique for dimensionality reduction here, since it doesn’t provide a mapping from the original space to the new space of reduced dimensions. If you run t-SNE on a dataset, you cannot take new examples and place them in the lower-dimensional space. – Paul Jun 8 '19 at 13:37
• And I also wouldn’t recommend jumping into auto-encoders. It’s true that PCA only addresses linear combinations between features, but remember that the correlation we are trying to remove is also linear. Before knowing more about the problem, such as if higher-order relations between features are important, I wouldn’t complicate the pipeline with anything beyond PCA. Sorry for being critical:-) – Paul Jun 8 '19 at 13:49
• I like the idea of trying different dimensionality reduction techniques for curiosity's sake. I shall think about using auto-encoders in particular; I feel like it is more a question of playing with my data between PCA and autoencoders to see which offers better performance in the imbalanced scenarios. I am also restricted to clustering as well, since I do not have training data. – imageimbalanceuser Jun 8 '19 at 19:05

The problem you are facing is that the majority of variance is explained by one class with the other class only making a small contribution. Dimesionality reduction via PCA is less than helpful since the focus there is on explaining "majority" of data.

One way, I suggest that you try out, is to first classify the data into more than 2 (for argument lets say 50) classes. This will force the unsupervised learning algorithm to account for "minority" factors. The collection of class centroids would be devoid of large part of the population variance and you can then run a classification on the centroids to identify the sub-classes of interest.