Clustering documents - how to evaluate results?

Question

I'm using DBSCAN clustering on a set of documents. The documents' content was converted to TF-IDF matrix, and I'd like to find consistent ways to evaluate the clusters when no added information is given (labels etc.).

1. Metrics comparing clusters - a score for each cluster - the goal here is to figure out which clusters are 'better' than others (more dense and more unique). Thought of two metrics:

   *Intra-cluster similarity - measure the similarity of documents inside each cluster - I used the cosine distances between documents in the TF-IDF representation

   *Inter-cluster similarity - measure the similarity between clusters (to highlight the more unique clusters) - similarly, planning to use the cosine distances between clusters' kernels

2. Metrics comparing clustering algorithms - a score for the clustering technique (such as Silhouette score). Will be used for hyper-parameter tuning and choosing an algorithm.

Any ideas?

Answer 1

None of the internal evaluation metrics will work well on text in my experience. Probably because of the curse of dimensionality.

Furthermore, DBSCAN does not cluster everything, but can also produce noise points. Few evaluation methods (and even fewer implementations...) handle it well that noise is not a cluster.