1
$\begingroup$

I would like to understand how to compute the accuracy of cluster analysis. I have hundred of texts. The dataset looks like as follows:

Author  Text Checked
Shakespeare ‘...’ yes
Keats ‘...’ no
...

In text there are some quotes from the author. Checked column has been filled in manually. What I would like to do is to group texts by authors. I already did it using kmeans and TfidfVectorizer. However I would like to understand if it is possible to test the accuracy of this approach. Do you know how I could do it? In case it would be not possible, could you please tell me what other approach I could take into account? It would be good to test different approach and see the level of accuracy. Any example it would be great.

$\endgroup$
1
$\begingroup$

There are different ways to address the task that you describe:

  • If the goal is simply to predict the author among a set of predefined authors, then this is not a clustering task (unsupervised) but a classification task (supervised). This implies that you would split the data between training and test set (or use cross-validation), train a model using the training set then apply it the test set, and finally evaluate the performance on the test set by comparing the predicted author with the true author for every text. Typically the performance is aggregated as f1-score for every author, then macro/micro f1-score.
  • If the goal is some kind of experiment to see whether a clustering algorithm can correctly identify the groups by author (spoiler: it certainly can't, at least not well), then you're doing unsupervised learning with an external evaluation. I'm not aware of any standard evaluation method for this context (there might be). I assume that you would need a matching method between the clusters ids and the authors (e.g. assign each author to the cluster which has the most instances by this author), then you can apply the regular classification evaluation.

Two additional points:

  • The general problem of distinguishing texts by their author is called stylometry. There are many variants, such as open or closed classification, author verification, author profiling, etc. It's an active area of NLP research (which means there's a lot of literature about it), and state of the art methods are quite different from regular classification/clustering. It doesn't meant that regular methods don't work at all, but usually not as well as specialized methods.
  • Features: TF-IDF words vectors are used for tasks where the "topic" of the text matters, for instance to classify articles about sports, politics, etc. There are better options for the features when the task is about writing style:
    • no focus on rare words (as IDF does), instead rare words should usually be discarded (because they cause massive overfitting)
    • words are not necessarily the right unit to consider. Character trigrams/4-grams have been proved to work better.
    • tokenizing can be tricky, it's often considered safer not to tokenize at all so that the author's punctuation habits are preserved as features.
| improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.