How evaluate text clustering?

Question

What metrics can be used for evaluating text clustering models? I used tf-idf + k-means, tf-idf + hierarchical clustering, doc2vec + k-means (metric is cosine similarity), doc2vec + hierarchical clustering (metric is cosine similarity). How to decide which model is the best?

Answer 1

Check out this paper. It also addresses question of how many clusters to use. The R package mclust has a routine which will try different cluster models/number of clusters and plot the Bayesian inference criterion (BIC). (great vignette here). It's a general method, meaning, something you can do without being domain/data specific. (It's always good to be domain-specific if you have the time and data.)

The chart is from the vignette by Lucca Scrucca. MClust tries 14 different clustering algorithms (represented by the different symbols), increasing the number of clusters from 1 to some default value. It's finds the BIC each time. Highest BIC is usually the best choice. You could apply this methodology to your own stable of clustering algorithms.

Answer 2

Check out silhouette score

Formula for i th data point

(b(i) - a(i)) / max(a(i),b(i))

where b(i) -> dissimilarity from nearest neighbouring cluster

a(i) -> dissimilarity between points within cluster

This gives a score between -1 and +1.

Interpretation

+1 means very good fit

-1 means misclassified [should have belonged to a different cluster]

After calculating silhouette score for each data point, you can take a call on the choice for the number of clusters.

Code Example

from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_samples, silhouette_score

X, y = make_blobs(n_samples=500,
                  n_features=2,
                  centers=4,
                  cluster_std=1,
                  center_box=(-10.0, 10.0),
                  shuffle=True,
                  random_state=1)  # For reproducibility

range_n_clusters = [2, 3, 4, 5, 6]

for n_clusters in range_n_clusters:
    # Initialize the clusterer with n_clusters value and a random generator
    # seed of 10 for reproducibility.
    clusterer = KMeans(n_clusters=n_clusters, random_state=10)
    cluster_labels = clusterer.fit_predict(X)

    # The silhouette_score gives the average value for all the samples.
    # This gives a perspective into the density and separation of the formed
    # clusters
    silhouette_avg = silhouette_score(X, cluster_labels)
    print("For n_clusters =", n_clusters,
          "The average silhouette_score is :", silhouette_avg)

    # Compute the silhouette scores for each sample
    sample_silhouette_values = silhouette_samples(X, cluster_labels)

Answer 3

A clustering quality measure would be very nice to have. Unfortunately, that measure is hard to calculate -- probably AI-hard. You are trying to reduce a very complex thing to a single number.

If it is AI-hard, then you could ask people to rate the clusterings somehow. It's not ideal, and won't scale but you will have a single number that represents something close to what you want.

Answer 4

Though silhouette_score is a good parameter to evaluate model performance, you can also use davies_bouldin_score. The Davies-Bouldin Score (DBS) can be imported from sklearn.metrics using sklearn library.

Here the minimum score is 0, but the maximum score can be above 1.

I would recommend saving the model-predicted output in excel and evaluating it manually to realize how the model predictions are coming.

Use:

from sklearn.metrics import davies_bouldin_score
dbs = davies_bouldin_score(feature_12.toarray(), model. Labels_)