0
$\begingroup$

In the preprint paper Text and code embeddings by contrastive pre-training, the authors describe a Transformer encoder which

maps the input, x and y, to embeddings, vx and vy respectively and the similarity between two inputs is quantified by the cosine similarity between their embeddings, vx and vy

And they state:

We found that using different delimiters leads to more stable training. For x, we use ‘[’ as [SOS]x and ‘]’ as [EOS]x, while we use ‘{’ and ‘}’ as [SOS]y and [EOS]y respectively for y

Is there an intuitive explanation for why using different delimiters is important for training stability?

$\endgroup$
0

1 Answer 1

0
$\begingroup$

Having different delimiters for the two segments allows the model to distinguish them. This may lead to the exploitation of correlations of the order with the label in the training data by the model.

This is just an hypothesis, of course.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.