New answers tagged

2

The distinction between supervised and unsupervised is a little bit tricky here. BERT pre-training is unsupervised with respect to the downstream tasks, but the pre-training itself is technically a supervised learning task. BERT is trained to predict words that have been masked in the input, so the target words are known at training time. The term ...


0

Unfortunately, there is little theoretical knowledge about what complex neural networks do. Transformers are known to be universal approximations, so in theory they can learn to do any function with the input sentence, unlike the other alternatives that you mention. Most of the time, the accuracy of the BERT-like model would be strictly better. In practice, ...


0

Here are the four modifications RoBERTa made for BERT: training the model longer, with bigger batches, over more data removing the next sentence prediction objective training on longer sequences dynamically changing the masking pattern applied to the training data. The authors also collect a large new dataset () of comparable size to other privately used ...


1

BERT models work with sentences, not words: the self-attention in the transformer architecture is considering each token in respect to all other tokens in the sentence. "I'm going to implement Transformers in Python" vs. "I'm going to watch Transformers on TV later tonight" - there should be enough surrounding context to distinguish them....


1

You don't need to make preprocessing as I understand, and the reason for this is that the Transformer makes an internal "dynamic" embedding of words that are not the same for every word; instead, the coordinates change depending on the sentence being tokenized due to the positional encoding it makes. Note the difference with Word2Vec, GloVe or ...


1

As far as I'm aware there's no perfect answer to this question. I agree with your analysis, the two options make sense: The first option corresponds to the correct labeling in theory, in the sense that it means exactly what one wants in this case: the words of the entity don't (necessarily) appear continuously. The second option makes things easier for the ...


0

To my understanding words unknown to the tokenizer will be masked with [UNKNOWN]. Your understanding is not correct. BERT's vocabulary is defined not at word level, but at subword level. This means that words may be represented as multiple subwords. The way subword vocabularies work mostly avoids having out-of-vocabulary words, because words can be divided ...


Top 50 recent answers are included