New answers tagged

1

The sentence "During pre-training, the model is trained on unlabeled data over different pre-training tasks." means that BERT was pre-trained on normal textual data on two tasks: masked language model (MLM) and next sentence prediction (NSP). There were no other classification/tagging labels present in the data, as the MLM predicts the text itself ...


2

Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary extraction process has also use lowercase text as input, and therefore gives as result a different vocabulary than the same vocabulary extraction process used ...


1

First, a clarification: normally, we distinguish tokenization and word segmentation. It is not clear to me what you exactly mean. Word segmentation is normally needed in languages without spaces between words, like Chinese. Tokenization is applied to most languages and is the process of splitting words in subword units to obtain a manageable vocabulary size ...


2

Subword tokenization is the norm nowadays in NLP models because: It mostly avoids the out-of-vocabulary (OOV) word problem. Word vocabularies cannot handle words that are not in the training data. This is a problem for morphologically-rich languages, proper nouns, etc. Subword vocabularies allow representing these words. By having subword tokens (and ...


Top 50 recent answers are included