1
$\begingroup$

I have a galaxy spectrum data set (total 22000). Similar to an electronic wave data, two dimensional (Flux vs Wavelength). A typical set of wavelength plot looks like below enter image description here

Now I am doing kmeans on this data set to cluster them based on their spectrum shape/pattern only (using sci-kit learn). Some results of the k means clustering is baffling me, I have made a flow chart of how the candidates clustered as I would go on increasing the number of clusters from k=2 to k=5. The flow chart graph of the division of candidates is below (this is on a smaller subset), enter image description here

Now my most basic question is why is the division of points in clusters happening like it is seen in the plot, as we traverse down the graph. More specifically, why for example in k=4 case in Cluster=1 it makes a group with such a mix bag ? couldnt kmeans instead select it's constituent 1078 or 580 candidates into a single (more cleaner) cluster ? Also is it a coincidence that there is always an identified 13 members group (in golden arrow) (or a 47).

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ It is an interesting way to visualize k-means. Since you have a 2D data, have you tried visualizing it something like the graph you see here? $\endgroup$
    – sai
    Oct 1, 2020 at 10:58
  • $\begingroup$ @sai , individual waves are of array 5000 length. So it is not a 2D data. Kmeans would cluster based on the wavelength distribution on this arrays. so it is a high dimenionsal data. Am I correct ? $\endgroup$
    – Ayan Mitra
    Oct 1, 2020 at 12:08
  • $\begingroup$ I do not think the length matters. Just the space in which your waves vary i.e., the min and max values of the flux and wavelength matter. $\endgroup$
    – sai
    Oct 1, 2020 at 12:13
  • $\begingroup$ How do you generate that chart? What do the individual arrows/colors mean? $\endgroup$
    – Ben Reiniger ♦
    Oct 1, 2020 at 13:56
  • $\begingroup$ You can't visualize directly in 2D as you are in 5000D however you can use a tool such as t-SNE to visualize your data points in 2D (try different parameters of perplexity) and check in the 2D space from t-SNE how are distributed clusters from KNN (@ sai I think KNN is not performed on indivual (time, value) points but rather on the whole signals, signals with similar shapes will be closer) $\endgroup$
    – mprouveur
    Oct 1, 2020 at 15:10

0

Reset to default