Im dealing with a dataset (text messages about source code comments) that are not labeled. I don't have a assumption about the implicits classes in this dataset. I want to discovery (by clustering) the common hidden patterns shared by the groups of messages. This is a unsupervised learning problem. I was asked how i will validate this method (patterns discovery, clusters) without a dataset of correct answers to measure the output of the model with the "reality". Im neither a specialist in the field of the messages dataset to manualy inspect and label the data. So, how to approach this question or provide a scientific explanation about the model output? How to prove that the clusters generated by the model are reasonable or correct?


In my opinion there are two ways:

  • Ask a few experts to assess the quality of the clusters based on a sample (after the clustering has been done, much easier than pre-annotating the whole data especially in the case of clustering)
  • If the clustering is done in the perspective of using the result in another task, the performance of this other task will reflect the quality of the clustering.

Imho any measure based on the distance between clusters or other technical measure would be a flawed evaluation, because it would depend on the quality of the representation. Such measures might provide some useful indications though, just not a proper evaluation for the task.

| improve this answer | |
  • 1
    $\begingroup$ +1 to keep resisting aginst wrong imagination of "Unsupervised learning can be evaluated without any additinal info/label" which is raising more and more in practice ... The fact that business managers need answer does not mean that we need to provide wrong answers. And it is happening more and more unfortunately $\endgroup$ – Kasra Manshaei Sep 23 '19 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.