New answers tagged transformer
0
Here are the four modifications RoBERTa made for BERT:
training the model longer, with bigger batches, over more data
removing the next sentence prediction objective
training on longer sequences
dynamically changing the masking pattern applied to the training data. The authors also collect a large new dataset () of comparable size to other privately used ...
0
I can recommend that you always look for any dataset or kaggle problem related to the particular solution that you want to carry out and see the architectures with the best results in the evaluation metrics of these datasets.
You can also check at https://paperswithcode.com/
where they have the papers tagged by type of problem and their link to github with ...
0
Either at training time or at inference time, both an LSTM and a Transformer decoder act exactly the same in terms of inputs and outputs:
At training time, you provide the whole sequence as input, and you obtain the next token predictions. In LSTMs, this training regime is called "teacher forcing"; we use this fancy name because LSTMs (RNNs in ...
0
First of all, I would not consider each letter as a token of your input sequence, think of the words as a whole as your tokens.
Regarding the problem of predicting the next token (word) given some input sequence, the accepted architecture nowadays is sequence-to-sequence with encoder-decoder, where you encode your input sequence (source sentence) into one or ...
1
You don't need to make preprocessing as I understand, and the reason for this is that the Transformer makes an internal "dynamic" embedding of words that are not the same for every word; instead, the coordinates change depending on the sentence being tokenized due to the positional encoding it makes.
Note the difference with Word2Vec, GloVe or ...
Top 50 recent answers are included