0
$\begingroup$

In the preprint paper Text and code embeddings by contrastive pre-training, the authors describe a Transformer encoder which

maps the input, x and y, to embeddings, vx and vy respectively and the similarity between two inputs is quantified by the cosine similarity between their embeddings, vx and vy

And they state:

We found that using different delimiters leads to more stable training. For x, we use ‘[’ as [SOS]x and ‘]’ as [EOS]x, while we use ‘{’ and ‘}’ as [SOS]y and [EOS]y respectively for y

Is there an intuitive explanation for why using different delimiters is important for training stability?

Improve this question
$\endgroup$
1
  • $\begingroup$ Hi @RobbiePalmer, welcome to the site. If you find the answers to your question useful, please consider upvoting them (once you have enough reputation). Also, please consider accepting one of the answers (with the tick mark ✓ next to it) if you consider it correct. $\endgroup$
    – noe
    May 3 at 14:37

1 Answer 1

Reset to default
0
$\begingroup$

Having different delimiters for the two segments allows the model to distinguish them. This may lead to the exploitation of correlations of the order with the label in the training data by the model.

This is just an hypothesis, of course.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.