1
$\begingroup$

GPT-1 mentions both Semi-supervised learning and Unsupervised pre-training but it seems like the same to me. Moreoever, "Semi-supervised Sequence Learning" of Dai and Le also more like self-supervised learning. So what the key differences between them?

$\endgroup$
1
$\begingroup$

Semi-supervised learning is having label for a fraction of data, but in self-supervised there is no label available. Imagine a huge question/answer dataset. No one labels that data but you can learn question answering right? Because you are able to retrieve relation between question and answer from data.

Or in modeling documents you need sentences which are similar and sentences which are dissimilar in order to learn document embedding but these detailed labels are usually not available. In this case you count sentences from same document as similar and sentences from two different documents as dissimilar and train your model (example idea: you can run a topic modeling on data and make similar/dissimilar labels more accurate). It is called self training.

$\endgroup$
5
  • $\begingroup$ Then can I consider the pre-training phase of BERT as 'self-supervised learning', and the fine-tuning phase of BERT as 'semi-supervised learning? $\endgroup$ May 28 at 8:32
  • $\begingroup$ I would call the fine-tuning part still "supervised" but you are right. In fine tuning you have a set of labeld data (however small) and you learn only from them so it is classic "supervised". Semi supervised means there are data in your training set for which you do not have labels. In general your high-level understanding is correct $\endgroup$ May 28 at 9:01
  • $\begingroup$ Got it. The overall process of BERT, ELMo, etc. can be considered as 'Semi-supervised learning' because they including both LM(self-supervised) and supervised learning. The training process of LM is self-supervised learning. Thank you. $\endgroup$ May 29 at 10:32
  • $\begingroup$ Welcome my friemd. If it answered your question you may consider accepting answer or upvoting it. $\endgroup$ May 29 at 10:51
  • $\begingroup$ Yeap definitely. Thank you $\endgroup$ May 29 at 12:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.