You first need to decide what your "label" is, and this label will apply to both your supervised and unsupervised models. Obviously, in a supervised world, knowing your label make training the model and evaluating it easy, so I won't go into that because it seems like you have a good handle.
In the unsupervised world, you'll have to think really hard about the mapping of the unsupervised outputs to the label you've decided to use for evaluation. A lot of this is going to come down to how you structure that unsupervised problem. Like, if you're going to be using unsupervised clustering, then maybe you run the model, see what the clusters look like, and see if there's a way to map those clusters the label you're evaluating.
It sounds like you're using unsupervised ranking and extracting top key phrases after the unsupervised ranking. In this case, you'd probably want to map the possible top key phrases to the labels you want to evaluate. Then after ranking and key phrase extraction, you can get a mapped "prediction label" based on the key phrases for each prediction. Of course, you'll likely have a messy collection of "prediction labels" because it's unlikely that each test prediction will output N-key phrases that perfectly match 1 label which means you'll have to come up with a way to round those maps (maybe you take the mode, maybe you round the average, it all depends on what you value most) to a singular prediction label. From there, doing recall and precision and f1 is straight forward.
I will say though, that trying to bend an unsupervised model to evaluation like a supervised model is only useful if recall, precision, and f1 are the best way to evaluate your model's usefulness in its problem space. This is very different from evaluating its effectiveness.
This question is: Does this models output fit the use case of the problem best? When I implement this output to an end user, would they appreciate this model more than another model? I'd encourage you to, on top of using classic evaluation scores to measure model effectiveness, also find a way to measure or understand model usefulness in the application that you have.
