New answers tagged

2 votes
Accepted

What is the difference between adding words to a tokenizer and training a tokenizer?

First, a clarification: tokenizers receive text and return tokens. These tokens may be words or not. Some tokenizers, for instance, return word pieces (i.e. subwords). This way, a single word may lead ...
user avatar
  • 16k
2 votes
Accepted

Smaller embedding size causes lower loss

New Answer The loss of a text generation task like question generation is normally the average categorical cross-entropy of the output at every time step. Drastically reducing the number of tokens ...
user avatar
  • 16k
0 votes
Accepted

What is the effect of the tokens?

The question is not precise enough, it depends on other factors: in general, a larger training set tends to lead to a better model. However it depends if the training set is really relevant and useful ...
user avatar
  • 22.2k

Top 50 recent answers are included