how to evaluate the combination of tfidf and kmeans

Question

For my nlp problem I'm using a combination of TFIDF and KMeans from the sklearn package. The tfidf gets the vectors and then I use Kmeans to cluster the texts based on the vectors. I have a few parameters of the TFIDF that I play around with like the n_gram, the input features, and the stop_words. The problem is how do I evaluate this model? My guess is I don't have to evaluate the KMeans model since its role is just to calculate the distances between points and that I just have to focus on the TFIDF model and parameters I end up using. Is this correct? If it is, how do I evaluate this model apart from manually shifting through the clusters to see if items are grouped correctly?

Edit: I forgot to mention: I don't have a target label. I'm basically grouping texts that look similar in clusters.

Answer 1

To evaluate a model based on K-means clustering and TF-IDF, you can use a variety of metrics.

About TF-IDF, you can use metrics from sklearn:

from sklearn import metrics

actual = predictionDF.select('label').toPandas()
predicted = predictionDF.select('prediction').toPandas()
print('accuracy score: {}%'.format(round(metrics.accuracy_score(actual,         predicted),3)*100))

One common metric is the silhouette coefficient, which measures how well each sample has been assigned to its cluster. This coefficient ranges from -1 to 1, with a high value indicating that the sample is well-matched to its cluster and a low value indicating that it is poorly matched.

https://stackoverflow.com/questions/40994347/sklearn-clustering-calculate-silhouette-coefficient-on-tf-idf-weigthed-data

You can also use other metrics such as the Calinski-Harabasz index or the Davies-Bouldin index, which measure the compactness and separation of the clusters. It's important to choose the right evaluation metric for your particular task and data.

For another unsupervised clustering, you can apply UMAP and TF-IDF. UMAP is a non-linear algorithm for dimensional reduction that works better than linear ones:

https://umap-learn.readthedocs.io/en/latest/document_embedding.html

In all cases, you need some target information, at least for a random sample (test data). Otherwise, by which basis could you evaluate if the results are correct or not?

See also:

https://subscription.packtpub.com/book/big-data-and-business-intelligence/9781788474221/7/ch07lvl1sec65/evaluating-tf-idf-model-performance

https://pyshark.com/calinski-harabasz-index-for-k-means-clustering-evaluation-using-python/