In the skip-gram model, the probability that a word $w$ is part of the set of context words $\{w_o^{(i)}\}$ $(i= 1:m)$ where $m$ is the context window around the central word, is given by:

$$p(w_o | w_c) = \frac{\exp{(\vec{u_o}\cdot \vec{v_c)}}}{\sum_{i\in V}\exp{(\vec{u_i}\cdot \vec{v_c)}}} $$

where $V$ is the number of words in the training set, $\vec{u_i}$ is the word embedding for the context word and $\vec{v_i}$ is the word embedding for the central word.

But this type of model is defining a linear transformation of the input similar to the one found in a multinomial logistic regression:

$$ p(y = c|\vec{x};\vec\theta) = \frac{\exp{(\vec{w_c}\cdot \vec{x})}}{\sum_{i \in N}\exp{(\vec{w_i}\cdot \vec x)}}$$

I understand that the real trick is in how you formulate the loss function, where in the skip-gram model instead of multiplying the probability of every class (every word) you just do it by a subset of words (the context). However, the transformations are linear instead of non-linear as I would expect if this was a neural network model.

I know that you can have some linear transformations in a DNN (actually linear composed to nonlinear composed to linear ...), but I thought the main purpose of using the term DNN and constructing a visual representation was that you had some non-linear transformations which if you choose carefully can be viewed as functions that range between -1,1 or 0,1 and it can be seen as "activation functions" which then induces this neural network graphical representation thing.

However, I fail to grasp this for word2vec and the skipgram model. Could anyone shed some light on this?


1 Answer 1


I think you are confused - the reason why Word2Vec is regarded as 'neural' is not due to its loss function, but that it uses neural network to estimate the word embedding ($\vec{u}$ and $\vec{v}$) (see section 2 of the original paper).

For example, I can have a ML problem with a loss function $L$ to minimize (on some data $X$ and target $y$). If I use a simple linear model to do the job, it is linear; or I would call it 'neural model' if I use (say) a CNN. Does not matter whether the loss $L$ is linear or else.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.