0
$\begingroup$

I am currently trying to understand the Paragraph Vector framework by reading the paper "Distributed Representation of Sentences and Documents" by Quoc Le and Thomas Mikolov but I have troubles following the formal description since my current understanding of Neural Networks is limited to mainly intuition. In the paper, they explain former techniques of learning word vector representations including Neural Language Models. Their formal description of the training task of Neural Language Models is as follows:

More formally, given a sequence of training words $w_1 , w_2 , w_3 , ..., w_T$ , the objective of the word vector model is to maximize the average log probability $\frac{1}{T}\overset{T-k}{\underset{t=k}{\sum}}\log p(w_t | w_{t-k}, ..., w_{t+k})$

Unfortunately, they don't explain why the goal is to maximize the average log probability and how they came up with this formula. Also search requests and looking through "A Neural Probabilistic Language Model" by Bengio et al. did not bring me any further.

My understanding of the training task is that given a context, the model predicts the missing word of a context. Moreover, out of all words in the vocabulary the word with the highest conditional probability given the context is selected as the missing word. Therefore I don't understand why this formula does not include some sort of $\mathrm{argmax}$ expression.

Any help on understanding why this is the goal and additional resources on this would be appreciated.

$\endgroup$

1 Answer 1

0
$\begingroup$

The problem is formulated as a classification problem over all possible next words. Cross entropy/log probs give you a smooth loss function. You don't use argmax for the same reason you don't train classification models with argmax.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.