I want to use BERT model for sentences similarity measuring task. I know that BERT models were trained with natural language inference architecture with dataset with labels neutral, entailment, contradiction.

My data to which I want to apply BERT for sentences similarity task has very specific terms and jargon, so I want to pretrain model on it before. But in that data there are only cases of entailment labels (about 20k rows). Is it a good idea to pretrain model on that data? How could I handle my problem the best way?

Thanks in advance

  • $\begingroup$ Can provide more details on the exact task? Is it semantic textual similarity or paraphrase identification or do you only do pre training and no supervised fine tuning? $\endgroup$
    – Jonathan
    Apr 24, 2022 at 15:26
  • $\begingroup$ @Sammy it is semantic similarity. i want to detect sentences which describe similar activities but with different words. isn't paraphrase identification based on semantic similarity too? $\endgroup$
    – Ir8_mind
    Apr 25, 2022 at 7:21
  • $\begingroup$ The difference is that Paraphrase Identification is a binary classification task while Semantic Textual Similarity is either a multi-categorical or regression task. Either way the approach would be similar as long as you take a supervised approach (i.e. have labeled data). $\endgroup$
    – Jonathan
    Apr 25, 2022 at 8:58


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.