I'm planning on building a basic content-based recommender system with word2vec and cosine similarity. The data consists of 300k documents in varying length.
How do I evaluate my model if I have no labels / categories whatsoever?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
If you're trying to create a content-based document recommender system, you want to measure success via some sort of ranking metric like precision@k.
But since you don't have user-document interaction histories, you're either going to have to make them yourself, or just do a bunch of document queries and see if they make sense.
If you're going to make user-document interaction histories yourself, I would just do 10-20 queries and go through the first 5 documents that get returned and label whether or not they match. Calculate precision@k for those results and now you have an idea of how you're doing.
Not sure if you're familiar with ranking metrics but the best way to look at it is to always compare to some baseline model. In your case, I would calculate precision@k for BoW, tfidf, LSA, and LDA with cosine similarity as other models to compare to.
Unfortunately not a ton of other options for the task of content-recommendation without interaction data to test on. But I also would add that just eyeballing the results a lot of the time will tell you how the model is doing.
-
$\begingroup$ Sorry, for the late reply. Do you have by any chance a scientific paper who gives me more insights on your method? $\endgroup$– jonasDec 18, 2020 at 16:23
-
$\begingroup$ No, what I said there was just based on my experience building recommender systems. Precision@k is a pretty straightforward metric, here is a medium post talking about it: medium.com/@m_n_malaeb/… As far as creating labels yourself, I think this is pretty straightforward but you just label whether or not you think that the document is relevant. Typically you would rely on actual user interactions data to get measures on this, but you don't have those so this is about the closest you will get to a scientific process. $\endgroup$– mkerrigDec 22, 2020 at 1:03
$\begingroup$
$\endgroup$
2
When you don't have label/categories then it's called Unsupervised Learning. You can solve this problem via Latent Dirichlet Allocation (LDA) model and then evaluate your model by splitting the texts in half and compare the topic assignment for each half using cosine similarity. The more similar the topic assignment, the better.
answered Nov 24, 2020 at 17:47
-
$\begingroup$ Why do I need LDA if I can use doc2vec and cosine similarity? The advantage is that I don't have to determine how many topics I have. Basically the only way to evaluate my model is by hand and through self defined tests? $\endgroup$– jonasNov 25, 2020 at 13:37
-
1$\begingroup$ Yes you can do that if you don't require topic. But then you can only evaluate your model using visualization like t-SNE plot $\endgroup$ Nov 25, 2020 at 14:01