Im dealing with a dataset (text messages about source code comments) that are not labeled. I don't have a assumption about the implicits classes in this dataset. I want to discovery (by clustering) the common hidden patterns shared by the groups of messages. This is a unsupervised learning problem. I was asked how i will validate this method (patterns discovery, clusters) without a dataset of correct answers to measure the output of the model with the "reality". Im neither a specialist in the field of the messages dataset to manualy inspect and label the data. So, how to approach this question or provide a scientific explanation about the model output? How to prove that the clusters generated by the model are reasonable or correct?
In my opinion there are two ways:
- Ask a few experts to assess the quality of the clusters based on a sample (after the clustering has been done, much easier than pre-annotating the whole data especially in the case of clustering)
- If the clustering is done in the perspective of using the result in another task, the performance of this other task will reflect the quality of the clustering.
Imho any measure based on the distance between clusters or other technical measure would be a flawed evaluation, because it would depend on the quality of the representation. Such measures might provide some useful indications though, just not a proper evaluation for the task.