8
$\begingroup$

What metrics can be used for evaluating text clustering models? I used tf-idf + k-means, tf-idf + hierarchical clustering, doc2vec + k-means (metric is cosine similarity), doc2vec + hierarchical clustering (metric is cosine similarity). How to decide which model is the best?

$\endgroup$
7
$\begingroup$

Check out this paper. It also addresses question of how many clusters to use. The R package mclust has a routine which will try different cluster models/number of clusters and plot the Bayesian inference criterion (BIC). (great vignette here). It's a general method, meaning, something you can do without being domain/data specific. (It's always good to be domain-specific if you have the time and data.)

The chart is from the vignette by Lucca Scrucca. MClust tries 14 different clustering algorithms (represented by the different symbols), increasing the number of clusters from 1 to some default value. It's finds the BIC each time. Highest BIC is usually the best choice. You could apply this methodology to your own stable of clustering algorithms.

enter image description here

| improve this answer | |
$\endgroup$
2
$\begingroup$

Check out silhouette score

Formula for i th data point

(b(i) - a(i)) / max(a(i),b(i))

where b(i) -> dissimilarity from nearest neighbouring cluster

a(i) -> dissimilarity between points within cluster

This gives a score between -1 and +1.

Interpretation

+1 means very good fit

-1 means misclassified [should have belonged to a different cluster]

After calculating silhouette score for each data point, you can take a call on the choice for the number of clusters.

Code Example

from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_samples, silhouette_score

X, y = make_blobs(n_samples=500,
                  n_features=2,
                  centers=4,
                  cluster_std=1,
                  center_box=(-10.0, 10.0),
                  shuffle=True,
                  random_state=1)  # For reproducibility

range_n_clusters = [2, 3, 4, 5, 6]

for n_clusters in range_n_clusters:
    # Initialize the clusterer with n_clusters value and a random generator
    # seed of 10 for reproducibility.
    clusterer = KMeans(n_clusters=n_clusters, random_state=10)
    cluster_labels = clusterer.fit_predict(X)

    # The silhouette_score gives the average value for all the samples.
    # This gives a perspective into the density and separation of the formed
    # clusters
    silhouette_avg = silhouette_score(X, cluster_labels)
    print("For n_clusters =", n_clusters,
          "The average silhouette_score is :", silhouette_avg)

    # Compute the silhouette scores for each sample
    sample_silhouette_values = silhouette_samples(X, cluster_labels)
| improve this answer | |
$\endgroup$
0
$\begingroup$

A clustering quality measure would be very nice to have. Unfortunately, that measure is hard to calculate -- probably AI-hard. You are trying to reduce a very complex thing to a single number.

If it is AI-hard, then you could ask people to rate the clusterings somehow. It's not ideal, and won't scale but you will have a single number that represents something close to what you want.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I don't think this is correct. I can simply feed a well-studied text document into the models. Then compare the cluster membership with my expectation. $\endgroup$ – SmallChess Oct 10 '16 at 11:20
  • $\begingroup$ Yep. Using "your" expectation is what you do when the measure is AI-hard. You would get a better measure if you include other people's expectations. $\endgroup$ – Ray Oct 10 '16 at 23:17
  • $\begingroup$ I have an idea. I can try to train classificator and fit it with labels from different models with same number of clusters. The better accurancy_score, the better model. $\endgroup$ – Толкачёв Иван Oct 11 '16 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.