36
votes
Accepted
What is the difference between model hyperparameters and model parameters?
Hyperparameters and parameters are often used interchangeably but there is a difference between them. You can call something a 'hyperparameter' if it cannot be learned within the estimator directly. ...
- 3,011
34
votes
Accepted
What is purpose of the [CLS] token and why is its encoding output important?
CLS stands for classification and its there to represent sentence-level classification.
In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog ...
- 5,609
33
votes
What is the difference between model hyperparameters and model parameters?
In addition to the answer above.
Model parameters are the properties of the training data that are learnt during training by the classifier or other ml model. For example in case of some NLP task: ...
- 2,147
21
votes
What is purpose of the [CLS] token and why is its encoding output important?
[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire ...
- 311
18
votes
What is purpose of the [CLS] token and why is its encoding output important?
In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks:
Masked language modeling: some random words are masked with [MASK]
token, the model ...
- 281
17
votes
What is the difference between model hyperparameters and model parameters?
Hyper-parameters are those which we supply to the model, for example: number of hidden Nodes and Layers,input features, Learning Rate, Activation Function etc in Neural Network, while Parameters are ...
- 171
15
votes
What is purpose of the [CLS] token and why is its encoding output important?
Here're my understandings:
(1)[CLS] appears at the very beginning of each sentence, it has a fixed embedding and a fix positional embedding, thus this token contains no information itself.
(2)However, ...
- 151
15
votes
Accepted
Is BERT a language model?
No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM.
A normal LM takes ...
- 22.2k
14
votes
ChatGPT's Architecture - Decoder Only? Or Encoder-Decoder?
Summary
ChatGPT is the fine-tuning of GPT-3.5, which is a language model based on a Transformer decoder with some modifications with respect to the original Transformer architecture. Therefore it is a ...
- 22.2k
13
votes
Word2Vec embeddings with TF-IDF
Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document ...
- 880
12
votes
Word2Vec embeddings with TF-IDF
Train a tfidfvectorizer with your corpus and use the following code:
tfidf = Tfidfvectorizer ()
dict(zip(tfidf.get_feature_names(), tfidf.idf_)))
Now you have a ...
11
votes
Accepted
What is whole word masking in the recent BERT model?
phil ##am #mon is a subword encoding of the single word “philammon” into 3 tokens. The comment just means that they mask words as opposed to tokens by taking into account subword encoding.
For more ...
- 456
9
votes
Are there any good out-of-the-box language models for python?
I also think that the first answer is incorrect for the reasons that @noob333 explained.
But also Bert cannot be used out of the box as a language model. Bert gives you the ...
- 413
9
votes
Accepted
What is the difference between GPT blocks and Transformer Decoder blocks?
GPT uses an unmodified Transformer decoder, except that it lacks the encoder attention part. We can see this visually in the diagrams of the Transformer model and the GPT model:
For GPT-2, this is ...
- 22.2k
8
votes
What is the difference between model hyperparameters and model parameters?
In machine learning, a model $M$ with parameters and hyper-parameters looks like,
$Y \approx M_{\mathcal{H}}(\Phi | D)$
where $\Phi$ are parameters and $\mathcal{H}$ are hyper-parameters. $D$ is ...
- 1,313
8
votes
Accepted
What does 'Linear regularities among words' mean?
By linear regularities among words, he meant that "Vectorized form of words should follow linear additive properties!"
V("King") - V("Man") + V("Woman") ~ V("Queen)
- 638
7
votes
Are there any good out-of-the-box language models for python?
I think the accepted answer is incorrect.
token.prob is the log-prob of the token being a particular type . I am guessing 'type' refers to something like POS-tag or type of named entity (it's not ...
- 71
6
votes
Are there any good out-of-the-box language models for python?
The spaCy package has many language models, including ones trained on Common Crawl.
Language model has a specific meaning in Natural Language Processing (NlP). A language model is a probability ...
- 20.2k
6
votes
Further Training a pre-trained LLM
Yes you are on the right track. What you are mentioning is called fine tuning the model. I personally have done this and used the same approach.
The LLM I used was GPT-J 6B to generate MCQ's. Some ...
- 1,866
5
votes
What is the difference between model hyperparameters and model parameters?
In simplified words,
Model Parameters are something that a model learns on its own.
For example,
1) Weights or Coefficients of independent variables in Linear regression model.
2) Weights or ...
5
votes
Accepted
What are the elements in a BERT word embedding?
Some points first:
BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which ...
- 6,512
5
votes
What size language model can you train on a GPU with x GB of memory?
Tldr; I’ve seen a good rule-of-thumb is about 14-18x times the model size for memory limits, so for a 10GB card, training your model would max out memory at roughly 540M parameters.
There is some ...
- 745
4
votes
Accepted
Neural Networks for Predictive typing
A neural network is in principle a good choice when you have A LOT of similar data and classification tasks. Predicting the next character (or word... which is just multiple characters) is such a ...
- 18.7k
4
votes
Accepted
How should I treat these non-English documents in the NLP task?
You can use these tips :
Should I exclude them for the corpus and from training the model?
You can do this if you don't have a lack of data. But I think excluding 500 docs from 30K docs won't make ...
- 2,140
4
votes
Accepted
Loss on whole sequences in Causal Language Model
The figure and the blog post are simply incorrect. Doing a reverse image search, I see that the image you posted comes from a blog post on Towards Data Science. That image is so wrong. Just think that ...
- 22.2k
4
votes
Do large pretrained language models already "know" about NLP tasks?
Large pretrained language models are empirically useful. They are empirically useful at prediction for established NLP benchmarks and novel tasks. Since this class of models is the best currently ...
- 20.2k
3
votes
how much text data is required for a meaningful use of word2vec
As word2vec is a neural network, it benefits from very large datasets. The Kaggle dataset is 50,000 reviews * ~5 sentences per review, so about a quarter million sentences. As they note, they get ...
- 1,205
3
votes
Accepted
How to calculate perplexity in PyTorch?
I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl ...
- 155
3
votes
How Exactly Does In-Context Few-Shot Learning Actually Work in Theory (Under the Hood), Despite only Having a "Few" Support Examples to "Train On"?
I highly recommend you read Microsoft's recent paper about In Context Learning. Although the focus is on LLM I think it can be generalised to other models.
The idea is to consider models as mesa|meta-...
- 61
3
votes
What is purpose of stacking N=6 blocks of encoder and decoder in transformer?
Does higher blocks represent longer phrases and learns what longer phrases attend to?
No, each layer can attend to arbitrarily long sequences.
While bottommost block represent single word and its ...
- 22.2k
Only top scored, non community-wiki answers of a minimum length are eligible