How does softmax work for vectors?

Question

In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a vector. So we have $V$ vectors in the output layer. And now we want to apply softmax to those vectors to get a vector of dimensionality $V$, where each component represents the probability of a word appearing with an input word. But how do we apply the softmax function to vectors? By definition the softmax function takes as input a single vector, but we have $V$ vectors each of a chosen dimensionality $N$.

Answer 1

I think there are some misunderstandings or, at least, some not-clear-naming problems in your question.

To train skipgram word embeddings, we take text data and, for each word, we try to predict the words in a window around it.

Each word is represented by an embedded vector of dimensionality $E$. In the embedding table, there are $V$ embedded vectors; each vector represents a word from the vocabulary, which is the list of $V$ words that will be supported by our embeddings. To create the vocabulary word list, usually, we select the $V$ most frequent words in our training data.

Our training data here are pairs of input word index and output word index, where output word index is one of the words in the context window of the input word. For each context window, we have multiple input-output entries in our training data.

The model to train the embeddings is just a matrix multiplication. We multiply the input embedding by a $E \times V$ matrix, obtaining a vector of dimensionality $V$. We then compute the softmax over this vector $V$ and then the negative log-likelihood loss.

Note that the softmax takes as input the vector of dimensionality $V$ and generates a vector of dimensionality $V$ with the probability distribution over the vocabulary words. In reality, softmax takes a batch of those vectors, but the softmax function is computed on each vector separately.

For reference, the original articles proposing the word2vec skipgram approach are Efficient Estimation of Word Representations in Vector Space and Distributed Representations of Words and Phrases and their Compositionality .