3
$\begingroup$

I started looking into word2vec and was wondering what the connection/difference between the NCE-Loss and the infoNCE-Loss is. I get the basic idea of both.

I have a hard time deriving one from another, do you have any idea ?

Thank you in advance!

$\endgroup$

1 Answer 1

1
$\begingroup$

Both NCE loss and InfoNCE loss use positive examples and sampled negative examples to contrast a true data distribution with a noise distribution. As a byproduct, you can learn something useful like $p(y|x)$ in the case of NCE, or high quality representations of your objects (e.g. documents, images, etc.) in the case of InfoNCE.

NCE casts the contrastive learning task as a binary classification problem, where we want to predict if a data point (a class and a context) came from the noise distribution or the true data distribution.

InfoNCE generalizes this idea and treats this task as a multiclass classification problem, where the goal is to predict which of the $N$ examples (1 positive and $N-1$ negatives) came from the true data distribution.

NCE uses the logistic function to generate probabilities, while InfoNCE generates probabilities from a calculation that looks more like softmax (i.e. $\frac{f(y, x)}{\sum_{i=1}^{N}f(y_i, x)}$). Once you have probabilities, you can apply cross-entropy loss to both.

Examining the loss for NCE and InfoNCE for a single example with 1 negative sample for both NCE and InfoNCE, we see that one loss function cannot easily be derived from the other. See this blog post for the loss functions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.