Search Results
| Search type | Search syntax |
|---|---|
| Tags | [tag] |
| Exact | "words here" |
| Author |
user:1234 user:me (yours) |
| Score |
score:3 (3+) score:0 (none) |
| Answers |
answers:3 (3+) answers:0 (none) isaccepted:yes hasaccepted:no inquestion:1234 |
| Views | views:250 |
| Code | code:"if (foo != bar)" |
| Sections |
title:apples body:"apples oranges" |
| URL | url:"*.example.com" |
| Saves | in:saves |
| Status |
closed:yes duplicate:no migrated:no wiki:no |
| Types |
is:question is:answer |
| Exclude |
-[tag] -apples |
| For more details on advanced search visit our help page | |
Results tagged with tokenization
Search options not deleted
user 133184
0
votes
1
answer
78
views
What is the difference between adding words to a tokenizer and training a tokenizer?
The title says it all. I was researching this question but couldn't find something useful. What is the difference between adding words to a tokenizer and training a tokenizer?
- 111
0
votes
0
answers
567
views
Slow and Fast tokenizer gives different outputs(sentencepiece tokenizer)
When i use T5TokenizerFast(Tokenizer of T5 arcitecture), the output is expected as follows:
['▁', '</s>', '▁Hello', '▁', '<sep>', '</s>']
But when i use normal tokenizer, it starts to split special t …
- 111
0
votes
1
answer
214
views
Smaller embedding size causes lower loss
When I convert my multilingual transformer model to a single lingual transformer model (got my languages embedding from the multilingual transformer and deleted other embeddings, decreased dimensions …
- 111
1
vote
0
answers
508
views
What are the inputs of encoder and decoder layers of transformer architecture?
In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the output of …
- 111