New answers tagged transformer
0
votes
BPE vs WordPiece Tokenization - when to use / which?
In contrast to BPE, WordPiece does not choose the most frequent symbol
pair, but the one that maximizes the likelihood of the training data
once added to the vocabulary.
So what does this mean exactly?...
- 1
0
votes
Creating class labels for custom DataSets efficiently (HuggingFace)
you can combine data of test and train in a single data frame.
Then you should split data frame using scalar test train split
- 141
2
votes
Accepted
Creating class labels for custom DataSets efficiently (HuggingFace)
This is a coding style issue, so people may well have different opinions! But I don't see any problem with the way you've coded it.
If you really want to reduce the number of lines of code you could ...
- 439
1
vote
What Preprocessing is Needed for Semantic Search Using Pre-trained Hugging Face Transformers?
Resumes are quite different from classic text because there are many proper nouns (names, companies, places, etc.) and other data difficult to classify (phone numbers, marks, age, etc.).
That's why ...
- 2,793
1
vote
Do I need training data in multiple languages for a multilingual transformer?
As far as I know, few multilingual models study their representation space to see if the representations of different languages occupy overlapping regions. The ones that do, usually find that the ...
- 16k
1
vote
Accepted
Transformers vs RNN basic doubt
There are multiple concepts mixed in your question.
Contextual vs. non-contextual word embeddings: word2vec is a non-contextual approach to obtaining token embeddings. This means that a specific word ...
- 16k
2
votes
Accepted
Smaller embedding size causes lower loss
New Answer
The loss of a text generation task like question generation is normally the average categorical cross-entropy of the output at every time step.
Drastically reducing the number of tokens ...
- 16k
1
vote
Transformers - Why Self Attention calculate dot product of q and k from of same word?
It is not necessarily the case. The matrics $K$ and $Q$ can be very different. The intuition is that these two projections allow the model to search for a particular piece of information in the hidden ...
- 1,406
0
votes
Accepted
What to do with Transformer Encoder output?
The typical approach for this is follow BERT's approach: add an extra special token at the beginning of the input sequence (in BERT it is [CLS]) and only use the ...
- 16k
Top 50 recent answers are included