1
$\begingroup$

I️ am working on a document classifier utilizing the Gensim libabry (doc2vec). After creating the model, I️ run the doc2vec.score() function on a document collection per document.

According to the Gensim api, the method returns an array of log probabilities, with values ranging from zero to negative infinity per document

My goal is to sort test documents of different lengths based on their probability to the model. I️ am new to log probabilities, and wasn’t sure if I️ need to sum, average, or grab the max value for my purposes.

How do I make use of the log probabilities?

$\endgroup$
  • $\begingroup$ I am not familiar with gensim. You are being returned an array of long probabilities per collection or per document, and if it is per collection I am assuming its collection size x embedding size. To get the probability of the document you can easily sum the probabilities for a document, thats the beauty of log transform. $\endgroup$ – Himanshu Rai Nov 13 '17 at 7:39
  • $\begingroup$ Just to clarify, the method returns an array of different lengths per document. So I️ wasn’t sure if it would be naive of me to add the log probabilities in the array and use this value for comparing or sorting amongst other documents values. $\endgroup$ – ldavila07 Nov 13 '17 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.