1
$\begingroup$

I have pandas table which contains data about different observations, each one was measured in different wavlength. These observsations are different than each other in the treatment they have gotten. The table looks something like this:

>>>name  treatment 410.1 423.2 445.6 477.1 485.2 ....
0 A1       0       0.01  0.02  0.04  0.05   0.87
1 A2       1       0.04  0.05  0.05  0.06  0.04
2 A3       2       0.03  0.02  0.03 0.01   0.03
3 A4       0       0.02  0.02  0.04  0.05  0.91
4 A5       1       0.05  0.06  0.04  0.05  0.02
...

I would like to classify the different observations based on their spectrum (the numerical columns).
I have tried to run PCA and to paint it according to the treatment the observations got, and to compare it to the results of classifications like k-means and Spectral clustering, but i'm not sure that I choose the right methods because is seems all the time like the clusters are too much like euclidean distance and i'm not sure that they take into account the spectrum (I have used all the numerical columns for the prediction).

This is for exampel the comparison between the PCA+Colors compared to the Spectral cllasification:
PCA: enter image description here

classification( the points located according to PCA1 PCA2 but the colores are according the the classification:
enter image description here

as you can see here, it seems like the classification is based on real distance and I would like something that take into account all the numerical values.

So, i'm looking for any insights regard other methods of classifications that could give me better results or maybe other ideas how I can check if there are clusters inside my data based on the measurments in different columns, like if I could predict the treatment from the clusters

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

This sounds like a normal supervised classification task.

Have you tried other standard methods like Support Vector Machines, RandomForests, Gradient Boosting, kNN, Neural Networks etc. as well or is there a particular reason why you only tried clustering methods.

Clustering methods like kmeans or spectral clustering are usually used in an unsupervised setting where class memberships are not available. Often they make certain assumptions about the data which might be violated, e.g. kmeans assumes spherical clusters, which is clearly not the case for your data.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I was afraid to use those method becuase for some days I have only 65 observations . I wanted to use the unsupervised necause I wanted to show that there are differences that can be identified in the different observations. I'm beginner in this so if you have any insight/my idea is not correct I would love to know :) $\endgroup$
    – Reut
    Sep 2, 2020 at 7:20
  • $\begingroup$ Like I said, try different supervised methods. If you have the labels available, why not make use of it? Regarding the limited amount of data, stick to simple methods, simple in the sense that they do not have a huge number of parameters (e.g. Neural Networks). This will prevent you from overfitting. $\endgroup$
    – Tinu
    Sep 2, 2020 at 8:48
  • $\begingroup$ Maybe I didn't explained my idea well: I wanted to use the unsupervisedd classification in order to be able to say : even when I don't teach the model what is the right answer, only telling him number of clusters , it guesses correct the different classes (like give the same class to tratment 2 ect. ) $\endgroup$
    – Reut
    Sep 2, 2020 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.