0
$\begingroup$

As the title says I'm trying to do clustering on a set of black and white images. These images are all 200x200 with black dots on a white canvas Example pics here (These are not actual photos from the data set. Just a general idea of what they look like). The idea is to hopefully find an underlying pattern when it comes to general shape of these images and hopefully cluster by that.

What I've done so far is turned each of my 200 x 200 image in a numpy array of size 40,000. Then put all the images together into a singly numpy array of 32k x 40k.

I'm kind of not sure where to go from there. What I did next was use scikit learn's TruncateSVD on my data set and set the paramater 'n_componenets=100' and fitted and transformed it, so now my data set of images are 32k x 100.

From here on out I don't know what's the best plan of action. Should I just start using k_means algos on my data set? and how would I visualize it?

Sorry if this is a lot. Any help/tips would be much appreciated it.

|improve this question
$\endgroup$
  • $\begingroup$ Please stop reposting the same question again and again in different sites... As said before, you need to do proper feature extraction and preprocessing. $\endgroup$ – Anony-Mousse Jul 23 at 23:04
  • $\begingroup$ Duplicate: datascience.stackexchange.com/q/56191/924 $\endgroup$ – Anony-Mousse Jul 24 at 5:36
0
$\begingroup$

As the dimension of your data is very high (40000), you can try using Spherical K-Means on your raw data. It uses the cosine similarity to compute the distance between 2 points. Indeed, euclidian distance is not so much informative in high dimensional vector spaces, because of the curse of dimensionality. see https://stats.stackexchange.com/questions/99171/why-is-euclidean-distance-not-a-good-metric-in-high-dimensions

Or you can apply a dimension reduction strategy like truncated SVD or ACP like you did, and then apply the classical k-means algorithm. So yeah you should try to apply k-means on your 100-dimensions data. Maybe try with k = 30 or something similar as 100 may already be too high.

|improve this answer
$\endgroup$
  • $\begingroup$ What should I do if I am not sure how many clusters there should be in my data set, or should I just try out random k values and see how it works. Last question would be how would I validate how accurate my program is? Would I be able to feed it an image and have it tell me what cluster it should be in? Sorry if these questions seem very basic $\endgroup$ – somedude1234 Jul 23 at 15:04
  • $\begingroup$ It is hard to determine the optimised number of clusters. You can maybe have an intuition by looking qualitatively the images, or if it is not possible you should empirically try several values. There exist several metrics to evaluate a given clustering result : en.wikipedia.org/wiki/Cluster_analysis#Internal_evaluation These are often based on the inter-distance between clusters (which you want to maximize) and intra-distance between points in a given cluster (which you want to minimize). $\endgroup$ – Alexis Pister Jul 23 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.