New answers tagged

0 votes

BPE vs WordPiece Tokenization - when to use / which?

In contrast to BPE, WordPiece does not choose the most frequent symbol pair, but the one that maximizes the likelihood of the training data once added to the vocabulary. So what does this mean exactly?...
user avatar
  • 1
0 votes

Creating class labels for custom DataSets efficiently (HuggingFace)

you can combine data of test and train in a single data frame. Then you should split data frame using scalar test train split
user avatar
  • 141
2 votes
Accepted

Creating class labels for custom DataSets efficiently (HuggingFace)

This is a coding style issue, so people may well have different opinions! But I don't see any problem with the way you've coded it. If you really want to reduce the number of lines of code you could ...
user avatar
  • 439
1 vote

What Preprocessing is Needed for Semantic Search Using Pre-trained Hugging Face Transformers?

Resumes are quite different from classic text because there are many proper nouns (names, companies, places, etc.) and other data difficult to classify (phone numbers, marks, age, etc.). That's why ...
user avatar
1 vote

Do I need training data in multiple languages for a multilingual transformer?

As far as I know, few multilingual models study their representation space to see if the representations of different languages occupy overlapping regions. The ones that do, usually find that the ...
user avatar
  • 16k
1 vote
Accepted

Transformers vs RNN basic doubt

There are multiple concepts mixed in your question. Contextual vs. non-contextual word embeddings: word2vec is a non-contextual approach to obtaining token embeddings. This means that a specific word ...
user avatar
  • 16k
2 votes
Accepted

Smaller embedding size causes lower loss

New Answer The loss of a text generation task like question generation is normally the average categorical cross-entropy of the output at every time step. Drastically reducing the number of tokens ...
user avatar
  • 16k
1 vote

Transformers - Why Self Attention calculate dot product of q and k from of same word?

It is not necessarily the case. The matrics $K$ and $Q$ can be very different. The intuition is that these two projections allow the model to search for a particular piece of information in the hidden ...
user avatar
  • 1,406
0 votes
Accepted

What to do with Transformer Encoder output?

The typical approach for this is follow BERT's approach: add an extra special token at the beginning of the input sequence (in BERT it is [CLS]) and only use the ...
user avatar
  • 16k

Top 50 recent answers are included