36 votes
Accepted

What is the difference between model hyperparameters and model parameters?

Hyperparameters and parameters are often used interchangeably but there is a difference between them. You can call something a 'hyperparameter' if it cannot be learned within the estimator directly. ...
enterML's user avatar
  • 3,031
34 votes
Accepted

What is purpose of the [CLS] token and why is its encoding output important?

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog ...
Noah Weber's user avatar
  • 5,669
33 votes

What is the difference between model hyperparameters and model parameters?

In addition to the answer above. Model parameters are the properties of the training data that are learnt during training by the classifier or other ml model. For example in case of some NLP task: ...
minerals's user avatar
  • 2,147
25 votes

What is purpose of the [CLS] token and why is its encoding output important?

[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire ...
Malgo's user avatar
  • 351
19 votes

What is purpose of the [CLS] token and why is its encoding output important?

In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks: Masked language modeling: some random words are masked with [MASK] token, the model ...
hoang tran's user avatar
19 votes

ChatGPT's Architecture - Decoder Only? Or Encoder-Decoder?

Summary ChatGPT is the fine-tuning of GPT-3.5, which is a language model based on a Transformer decoder with some modifications with respect to the original Transformer architecture. Therefore it is a ...
noe's user avatar
  • 26.5k
17 votes

What is the difference between model hyperparameters and model parameters?

Hyper-parameters are those which we supply to the model, for example: number of hidden Nodes and Layers,input features, Learning Rate, Activation Function etc in Neural Network, while Parameters are ...
Lakshmi Prasad Y's user avatar
15 votes

What is purpose of the [CLS] token and why is its encoding output important?

Here're my understandings: (1)[CLS] appears at the very beginning of each sentence, it has a fixed embedding and a fix positional embedding, thus this token contains no information itself. (2)However, ...
BigMoyan's user avatar
  • 151
15 votes
Accepted

Is BERT a language model?

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes ...
noe's user avatar
  • 26.5k
13 votes

Word2Vec embeddings with TF-IDF

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document ...
Maxim's user avatar
  • 890
12 votes

Word2Vec embeddings with TF-IDF

Train a tfidfvectorizer with your corpus and use the following code: tfidf = Tfidfvectorizer () dict(zip(tfidf.get_feature_names(), tfidf.idf_))) Now you have a ...
Aayush Shrivastav's user avatar
11 votes
Accepted

What is whole word masking in the recent BERT model?

phil ##am #mon is a subword encoding of the single word “philammon” into 3 tokens. The comment just means that they mask words as opposed to tokens by taking into account subword encoding. For more ...
BookYourLuck's user avatar
9 votes

Are there any good out-of-the-box language models for python?

I also think that the first answer is incorrect for the reasons that @noob333 explained. But also Bert cannot be used out of the box as a language model. Bert gives you the ...
lads's user avatar
  • 413
9 votes
Accepted

What is the difference between GPT blocks and Transformer Decoder blocks?

GPT uses an unmodified Transformer decoder, except that it lacks the encoder attention part. We can see this visually in the diagrams of the Transformer model and the GPT model: For GPT-2, this is ...
noe's user avatar
  • 26.5k
8 votes

What is the difference between model hyperparameters and model parameters?

In machine learning, a model $M$ with parameters and hyper-parameters looks like, $Y \approx M_{\mathcal{H}}(\Phi | D)$ where $\Phi$ are parameters and $\mathcal{H}$ are hyper-parameters. $D$ is ...
Dynamic Stardust's user avatar
8 votes
Accepted

What does 'Linear regularities among words' mean?

By linear regularities among words, he meant that "Vectorized form of words should follow linear additive properties!" V("King") - V("Man") + V("Woman") ~ V("Queen)
Preet's user avatar
  • 638
7 votes

Are there any good out-of-the-box language models for python?

I think the accepted answer is incorrect. token.prob is the log-prob of the token being a particular type . I am guessing 'type' refers to something like POS-tag or type of named entity (it's not ...
noob333's user avatar
  • 71
7 votes

Further Training a pre-trained LLM

Yes you are on the right track. What you are mentioning is called fine tuning the model. I personally have done this and used the same approach. The LLM I used was GPT-J 6B to generate MCQ's. Some ...
spectre's user avatar
  • 2,065
7 votes
Accepted

Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?

There are many contributing factors to the abundance of research based on BERT vs the research based on Llama: Age: BERT has been around for far longer than Llama (2018 vs 2023), so it has more ...
noe's user avatar
  • 26.5k
6 votes

Are there any good out-of-the-box language models for python?

The spaCy package has many language models, including ones trained on Common Crawl. Language model has a specific meaning in Natural Language Processing (NlP). A language model is a probability ...
Brian Spiering's user avatar
6 votes
Accepted

What size language model can you train on a GPU with x GB of memory?

Tldr; I’ve seen a good rule-of-thumb is about 14-18x times the model size for memory limits, so for a 10GB card, training your model would max out memory at roughly 540M parameters. There is some ...
brewmaster321's user avatar
5 votes

What is the difference between model hyperparameters and model parameters?

In simplified words, Model Parameters are something that a model learns on its own. For example, 1) Weights or Coefficients of independent variables in Linear regression model. 2) Weights or ...
Manju Savanth's user avatar
5 votes
Accepted

What are the elements in a BERT word embedding?

Some points first: BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which ...
Kasra Manshaei's user avatar
4 votes
Accepted

How should I treat these non-English documents in the NLP task?

You can use these tips : Should I exclude them for the corpus and from training the model? You can do this if you don't have a lack of data. But I think excluding 500 docs from 30K docs won't make ...
Shubham Panchal's user avatar
4 votes
Accepted

Loss on whole sequences in Causal Language Model

The figure and the blog post are simply incorrect. Doing a reverse image search, I see that the image you posted comes from a blog post on Towards Data Science. That image is so wrong. Just think that ...
noe's user avatar
  • 26.5k
4 votes

Do large pretrained language models already "know" about NLP tasks?

Large pretrained language models are empirically useful. They are empirically useful at prediction for established NLP benchmarks and novel tasks. Since this class of models is the best currently ...
Brian Spiering's user avatar
3 votes
Accepted

Words to numbers faster lookup

I'd like to extend the great @Emre's answer with another example - we are going to replace all tokenized words from the "1984" (c) George Orwell (120K words): ...
MaxU - stand with Ukraine's user avatar
3 votes

how much text data is required for a meaningful use of word2vec

As word2vec is a neural network, it benefits from very large datasets. The Kaggle dataset is 50,000 reviews * ~5 sentences per review, so about a quarter million sentences. As they note, they get ...
j.a.gartner's user avatar
  • 1,215
3 votes
Accepted

How to calculate perplexity in PyTorch?

I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl ...
Faruk's user avatar
  • 155
3 votes

How Exactly Does In-Context Few-Shot Learning Actually Work in Theory (Under the Hood), Despite only Having a "Few" Support Examples to "Train On"?

I highly recommend you read Microsoft's recent paper about In Context Learning. Although the focus is on LLM I think it can be generalised to other models. The idea is to consider models as mesa|meta-...
Xmaster6y's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible