Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have just built a simple doc2vec model using the gensim library, pretty much followed the tutorial located here.
The methods provided for checking the quality of the model are very manual and require reading the similar docs, is there a way to calculate some other metrics from the model to try and improve its performance?
Evaluation is based on the task, not the type of model used for it. In the tutorial that you link the task would be simple document similarity. Afaik a more common variant is the information retrieval setting, where the goal is to rank documents according to their similarity against a query document.
The tutorial mentions that they use a test set for which a link to a paper is provided. This paper explains that human annotators rated the similarity level between pairs of documents. That would be the standard way to obtain annotated data, but of course it takes a lot of manpower. It doesn't seem that the ground truth information is provided with the test set used in the tutorial though, I assume that this is why the author only proposes a manual inspection of the results.
In general there are probably many benchmark datasets (with ground truth annotations) publicly available for similar tasks, but they are not always easy to find and I'm not very knowledgeable about these. I think a good place to start would be the SemEval series of competitions, every year they release various datasets related to this kind of task.
