CLIP Paper or CLAP Paper (I dont understand the loss function) - Can you help?

Question

Can somebody help me understand this Contrastive Learning pretraining paper? This explanation comes from https://arxiv.org/pdf/2206.04769.pdf (page 2). I understand that they apply an audio encoder to the signal and a text encoder to the text (equation 1) then they map both encoder into the same size through a linear layer (equation two) and after since both learned linear projections have the same dimension, you can now compute the dot product or cosine similarity just as they are doing in equation (3) with a temperature factor “Tau”. My question is here: So far C (equation 3) is an NxN similarity matrix between the audio and text representation. I guess this matrix is symmetric right? In equation (4) they compute the average of the “audio loss” and the “text loss” and they say “l_k = 1/N sum ( log (diag (softmax C) , but I ‘m seeing that C is the same for Loss_text and Loss_audio ; why are they computing them twice? Are they the same? I’ve heard that in contrastive pretraining your loss should put every similar elements in a close space as well as dissimilar spaces far from each other, if this is considering only the diagonal matrix how is this actually happening? Well there are a lot of questions but if you can help with at least one of them that’s a huge advance for me.

Answer 1

Start with a batch of (audio, text) pairs denoted $( X_a, X_t )$.

The text data is passed through a text encoder $ f_t $ to produce a text embedding $ \hat{X}_t = f_t(X_t) $.

The audio data is passed through an audio encoder $ f_a $ to produce an audio embedding $ \hat{X}_a = f_a(X_a) $.

The embeddings $ (\hat{X}_t, \hat{X}_a) $ are different dimensions, so we pass each of them through a linear transformation to project them to the same dimensionality. This gives $ E_t = L_t(\hat{X}_t) $ and $ E_a = L_a(\hat{X}_a) $. The paper forgets the hat notation in equation 2.

Then we do a pair-wise comparison of all elements in $ E_a $ against all elements in $ E_t $ via $ C = \tau * (E_t • E_a^\top) $

$ C $ is an $ N \times N $ matrix ($N$ being the batch size) where the rows represent text items and columns represent audio items.

We then define our loss function $ \ell_k = \dfrac{1}{N} \sum_{i=0}^{N} log(diag(softmax(C))$ and compute our loss $\mathcal{L} = 0.5 * (\ell_{text}(C) + \ell_{audio}(C))$

Now do your question - how is $\ell_{text}(C)$ different from $\ell_{audio}(C)$, and are we doing the same computation twice?

The answer is no because we are computing the softmax along different dimensions. Note the description along text and audio axis respectively.

When we compute $\ell_{text}(C)$, we compute the softmax along the rows of $C$, while $\ell_{audio}(C)$ computes the softmax along the columns of $C$.

Now to your next question about only considering the diagonal elements.

All the off-diagonal elements contribute to the softmax calculation. The loss basically says "make the softmax term along the main diagonal really big". To do this, the model has to increase the similarity of items on the diagonal (bringing the same items closer together), but also decrease the similarity of items off the diagonal (moving different items farther apart). The softmax along a row/column of $C$ accomplishes the contrastive task of maximizing/minimizing similarity between paired/unpaired items.