Through performing clustering on a set of 1,000,000 text documents, I have identified 100 clusters. I am particularly interested in, say, 10 of the clusters. Imagine, I now have an additional set of 100,000 documents (not part of the initial 1 mil documents). I wonder if there is a way to efficiently check whether each one of the 100K new documents belongs to one of the 10 clusters.
1 Answer
$\begingroup$
$\endgroup$
- First Idea: Go as usual. Calculate the distance according to of each new document to the centroids of 100 clusters and see to which cluster the belong (minimum distance wins).
- Second Idea: You already know which clusters are your targets. Assign a label (e.g. 1) to these 10 and and another label (0) to the rest 90. Then train a classifier and try to predict the label for new documents. (The process of labeling in supervised learning is usually based on expert annotation i.e. an acceptable accuracy in labeling. But with this method you lose that accuracy so expect that not all the labels are perfectly fine as a clustering algorithm is blindly do that)
answered Oct 23, 2018 at 9:55