Timeline for How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>
Current License: CC BY-SA 4.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 29, 2020 at 10:16 | vote | accept | Kevin Ling | ||
| Jan 20, 2020 at 9:46 | comment | added | noe | I updated the answer with information regarding the blog post you linked. | |
| Jan 20, 2020 at 9:45 | history | edited | noe | CC BY-SA 4.0 |
added 422 characters in body
|
| Jan 18, 2020 at 11:50 | comment | added | Kevin Ling | @ncasas Here is the post, minimaxir.com/2019/09/howto-gpt2. Also, in the library gpt-2-simple, it also use the token like <|startoftext|>, any thoughts on it? | |
| Jan 15, 2020 at 10:33 | comment | added | noe | @Aditya if you are finetuning BERT on data that use those special tokens for such specific purposes, it may work. If you are taking BERT's weights as is and expect that using those tokens in different ways from what BERT was trained on, I would not expect good results. | |
| Jan 14, 2020 at 16:24 | comment | added | Aditya | A bit off topic ncasas, What's your views on adding auxillary token's (like SEP,CLS etc) as some kind of relevant feats presence being encoded into Bert? e.g. [is_something_specific_present] etc | |
| Jan 14, 2020 at 6:44 | comment | added | noe | @KevinLing please link to the posts you are referring to if you want to have further feedback on their approach. | |
| Jan 14, 2020 at 6:43 | comment | added | noe |
BERT natively supports receiving 2 sentences separated by the token [SEP], but this is used for the next sentence classification task.
|
|
| Jan 14, 2020 at 3:47 | comment | added | Kevin Ling | Or if another question, other than the normal punctuation, is there any way I can add in some special sentence separator? | |
| Jan 14, 2020 at 3:46 | comment | added | Kevin Ling | As I read from some technical blogs, typically they will add these tokens as a sentence separator. So I am confused now | |
| Jan 13, 2020 at 22:26 | comment | added | noe | Extending the vocabulary of an already trained model is normally not a good idea (apart from being technically challenging due to the differences in tensor sizes). Also, your examples of special tokens don't add anything new, so I see no point in trying hard to keep them. | |
| Jan 13, 2020 at 16:36 | comment | added | Aditya | Or we can also extend the vocab to add the new token's depending on the task? | |
| Jan 13, 2020 at 13:13 | history | edited | noe | CC BY-SA 4.0 |
added 1 character in body
|
| Jan 13, 2020 at 10:45 | history | answered | noe | CC BY-SA 4.0 |