1
$\begingroup$

I have just built a simple doc2vec model using the gensim library, pretty much followed the tutorial located here.

The methods provided for checking the quality of the model are very manual and require reading the similar docs, is there a way to calculate some other metrics from the model to try and improve its performance?

$\endgroup$
  • $\begingroup$ Calculating metrics from the model is never a proper evaluation method, because it can only be based on what the model already knows (or thinks it knows). It's the equivalent of a student grading their own test. So I'm afraid you'll probably need some annotated data. $\endgroup$ – Erwan Aug 8 at 21:46
  • $\begingroup$ @Erwan thanks! Could you provide an example of what annotation you would add to data to verify a doc2vec model? $\endgroup$ – dendog Aug 9 at 7:17
1
$\begingroup$

Evaluation is based on the task, not the type of model used for it. In the tutorial that you link the task would be simple document similarity. Afaik a more common variant is the information retrieval setting, where the goal is to rank documents according to their similarity against a query document.

The tutorial mentions that they use a test set for which a link to a paper is provided. This paper explains that human annotators rated the similarity level between pairs of documents. That would be the standard way to obtain annotated data, but of course it takes a lot of manpower. It doesn't seem that the ground truth information is provided with the test set used in the tutorial though, I assume that this is why the author only proposes a manual inspection of the results.

In general there are probably many benchmark datasets (with ground truth annotations) publicly available for similar tasks, but they are not always easy to find and I'm not very knowledgeable about these. I think a good place to start would be the SemEval series of competitions, every year they release various datasets related to this kind of task.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.