Image Embeddings - Negative Sampling and Imbalanced Class Issues

Question

I am using the negative sampling approach used in Word2Vec to train some image embeddings. From what I have read, for every positive example, we are creating a number of negative examples.

Question: Why do we use an imbalanced dataset here? Presumably we will get the normal issue where the algorithm ends up predicting the negative label to minimise the cost function? I understand that the aim isn't really to use it as a prediction model, but rather to extract the embeddings, but what is the benefit of having an imbalanced class here?

Answer 1

I don't think we are creating a number of negative examples for every positive example. Negative sampling is done to efficiently compute the softmax.

Word2vec tries to maximize the similarity for words in similar contexts. For e.g. given drink the NN should predict water with maximum probability.

Suppose you have V (typically > 1 million) words in your training data, so the last layer of your model has V neurons. For every word in your training data, you would have to compute output from all the V neurons (to compute softmax). It is computationally very expensive. Negative sampling is one way to address this problem. Instead of computing the all the V outputs, we just sample few words and approximate the softmax.

Negative sampling can be used to speed up neural networks where the number of output neurons is very high. Hierarchical softmax is another technique that's used for training word2vec.

Answer 2

Reading through the theoretical part of the tutorial, I understand that ideally, in the Word2Vec Kares architecture, for each target word, you would input all the available context words in order to make the network learn (through reinforcement) the words typically appearing in the same context as the target words (the "positive examples" in your question) and reduce the similarity between the target word and words not typically appearing in its context (the "negative examples" in your question).

You typically go through all the "positive examples" (they are not many) so that the corresponding words end up being placed in the same region of the 300-D vector space. However, going through all the "negative examples" would be a big computational burden due to their sheer amount (they are too many of them). So you "randomly" select some of them (the "negative samples") and train only for them.

So indeed there is some "imbalance" (you select "some" instead of "all" the "negative examples" for each target word) since the method makes sure that the target word is more "similar" to words of the same context in comparison to some (instead of all) the words of different contexts, but the same imbalance gives a substantial computational gain. After you have completed the training, you end up with a very nice 300-D representation of your words, readily available in the embedding layer, effectively solving the word embedding problem.

See also this answer from Quora.