1
$\begingroup$

I️ am working on a document classifier utilizing the Gensim libabry (doc2vec). After creating the model, I️ run the doc2vec.score() function on a document collection per document.

According to the Gensim api, the method returns an array of log probabilities, with values ranging from zero to negative infinity per document

My goal is to sort test documents of different lengths based on their probability to the model. I️ am new to log probabilities, and wasn’t sure if I️ need to sum, average, or grab the max value for my purposes.

How do I make use of the log probabilities?

Improve this question
$\endgroup$
2
  • $\begingroup$ I am not familiar with gensim. You are being returned an array of long probabilities per collection or per document, and if it is per collection I am assuming its collection size x embedding size. To get the probability of the document you can easily sum the probabilities for a document, thats the beauty of log transform. $\endgroup$
    – Himanshu Rai
    Nov 13, 2017 at 7:39
  • $\begingroup$ Just to clarify, the method returns an array of different lengths per document. So I️ wasn’t sure if it would be naive of me to add the log probabilities in the array and use this value for comparing or sorting amongst other documents values. $\endgroup$
    – ldavila07
    Nov 13, 2017 at 14:56

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.