1
$\begingroup$

I am working on an ML supervised learning project for Key-phrase extraction. The evaluation dataset (different from training-validation datasets) contains around 200 phrases labeled as key or not key.

The problem is that I want to compare the results obtained (in terms of precision, recall and f1) via different classifier's algorithms with existing unsupervised methods. These methods are based on ranking and extract the top-k key phrases.

Using the dataset that I mention, what is the proper way to calculate precision and recall for these unsupervised ranking methods?

$\endgroup$

1 Answer 1

2
$\begingroup$

You first need to decide what your "label" is, and this label will apply to both your supervised and unsupervised models. Obviously, in a supervised world, knowing your label make training the model and evaluating it easy, so I won't go into that because it seems like you have a good handle.

In the unsupervised world, you'll have to think really hard about the mapping of the unsupervised outputs to the label you've decided to use for evaluation. A lot of this is going to come down to how you structure that unsupervised problem. Like, if you're going to be using unsupervised clustering, then maybe you run the model, see what the clusters look like, and see if there's a way to map those clusters the label you're evaluating.

It sounds like you're using unsupervised ranking and extracting top key phrases after the unsupervised ranking. In this case, you'd probably want to map the possible top key phrases to the labels you want to evaluate. Then after ranking and key phrase extraction, you can get a mapped "prediction label" based on the key phrases for each prediction. Of course, you'll likely have a messy collection of "prediction labels" because it's unlikely that each test prediction will output N-key phrases that perfectly match 1 label which means you'll have to come up with a way to round those maps (maybe you take the mode, maybe you round the average, it all depends on what you value most) to a singular prediction label. From there, doing recall and precision and f1 is straight forward.

I will say though, that trying to bend an unsupervised model to evaluation like a supervised model is only useful if recall, precision, and f1 are the best way to evaluate your model's usefulness in its problem space. This is very different from evaluating its effectiveness.

This question is: Does this models output fit the use case of the problem best? When I implement this output to an end user, would they appreciate this model more than another model? I'd encourage you to, on top of using classic evaluation scores to measure model effectiveness, also find a way to measure or understand model usefulness in the application that you have.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.