Difference between NCE-Loss and InfoNCE-Loss

Question

I started looking into word2vec and was wondering what the connection/difference between the NCE-Loss and the infoNCE-Loss is. I get the basic idea of both.

I have a hard time deriving one from another, do you have any idea ?

Thank you in advance!

Answer 1

Both NCE loss and InfoNCE loss use positive examples and sampled negative examples to contrast a true data distribution with a noise distribution. As a byproduct, you can learn something useful like $p(y|x)$ in the case of NCE, or high quality representations of your objects (e.g. documents, images, etc.) in the case of InfoNCE.

NCE casts the contrastive learning task as a binary classification problem, where we want to predict if a data point (a class and a context) came from the noise distribution or the true data distribution.

InfoNCE generalizes this idea and treats this task as a multiclass classification problem, where the goal is to predict which of the $N$ examples (1 positive and $N-1$ negatives) came from the true data distribution.

NCE uses the logistic function to generate probabilities, while InfoNCE generates probabilities from a calculation that looks more like softmax (i.e. $\frac{f(y, x)}{\sum_{i=1}^{N}f(y_i, x)}$). Once you have probabilities, you can apply cross-entropy loss to both.

Examining the loss for NCE and InfoNCE for a single example with 1 negative sample for both NCE and InfoNCE, we see that one loss function cannot easily be derived from the other. See this blog post for the loss functions.