In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a vector. So we have $V$ vectors in the output layer. And now we want to apply softmax to those vectors to get a vector of dimensionality $V$, where each component represents the probability of a word appearing with an input word. But how do we apply the softmax function to vectors? By definition the softmax function takes as input a single vector, but we have $V$ vectors each of a chosen dimensionality $N$.

enter image description here


1 Answer 1


I think there are some misunderstandings or, at least, some not-clear-naming problems in your question.

To train skipgram word embeddings, we take text data and, for each word, we try to predict the words in a window around it.

enter image description here

Each word is represented by an embedded vector of dimensionality $E$. In the embedding table, there are $V$ embedded vectors; each vector represents a word from the vocabulary, which is the list of $V$ words that will be supported by our embeddings. To create the vocabulary word list, usually, we select the $V$ most frequent words in our training data.

Our training data here are pairs of input word index and output word index, where output word index is one of the words in the context window of the input word. For each context window, we have multiple input-output entries in our training data.

The model to train the embeddings is just a matrix multiplication. We multiply the input embedding by a $E \times V$ matrix, obtaining a vector of dimensionality $V$. We then compute the softmax over this vector $V$ and then the negative log-likelihood loss.

Note that the softmax takes as input the vector of dimensionality $V$ and generates a vector of dimensionality $V$ with the probability distribution over the vocabulary words. In reality, softmax takes a batch of those vectors, but the softmax function is computed on each vector separately.

For reference, the original articles proposing the word2vec skipgram approach are Efficient Estimation of Word Representations in Vector Space and Distributed Representations of Words and Phrases and their Compositionality .

  • $\begingroup$ Hmm... I am confused. "Each word is represented by an embedded vector of dimensionality 𝐸". I thought the whole purpose of skipgram is to find an embedded vector of each word? I though the input is a one hot encoding of a word, the hidden layer is a vector of dimensionality of the embedded vector and then the output is a list of vectors as seen in the picture I added $\endgroup$
    – Ruediger
    May 2, 2023 at 12:16
  • $\begingroup$ One-hot encoding + matrix multiplication is an equivalent but suboptimal way of implementing an embedding layer. $\endgroup$
    – noe
    May 2, 2023 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.