9 votes
Accepted

Does fine-tuning require retraining the entire model?

No, you don't need to retrain the entire model. Fine-tuning refers to taking the weights trained in the general model and then continuing training a bit using your specific data. Using this approach, ...
  • 537
7 votes
Accepted

How is GPT able to handle large vocabularies?

GPT-2 does not use a word-level vocabulary but a subword-level vocabulary, specifically byte-pair encoding (BPE). This means that it does not predict the next word but the next subword token. BPE ...
  • 17.9k
6 votes

What's the right input for gpt-2 in NLP

Updated answer After reading @Jessica's answer, I carefully read the original GPT-2 paper and I confirm that the authors do not add special tokens, but simply the text ...
  • 17.9k
6 votes

Does BERT has any advantage over GPT3?

This article on Medium introduces GPT-3 makes some comparisons with BERT. Specifically, section 4 examines how GPT-3 and BERT differ and mentions that: "On the Architecture dimension, BERT still ...
  • 161
5 votes

Does fine-tuning require retraining the entire model?

Yes. If open-sourced, we will be able to customize the model to our requirements. This is one of the most important modelling techniques called Transfer Learning A pre-trained model, such as GPT-3, ...
4 votes

Does BERT has any advantage over GPT3?

BERT needs to be fine-tuned to do what you want. GPT-3 cannot be fine-tuned (even if you had access to the actual weights, fine-tuning it would be very expensive) If you have enough data for fine-...
  • 141
4 votes
Accepted

Can I fine tune GPT-3?

The weights of GPT-3 are not public. You can fine-tune it but only through the interface provided by OpenAI. In any case, GPT-3 is too large to be trained on CPU. About other similar models, like GPT-...
  • 17.9k
4 votes
Accepted

How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>

BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained ...
  • 17.9k
3 votes

BERT vs GPT architectural, conceptual and implemetational differences

To start with your last question: you correctly say that BERT is an encoder-only model trained with the masked language-modeling objective and operates non-autoregressively. GPT-2 is a decode-only ...
  • 1,501
3 votes

What exactly are the parameters in GPT-3's 175 billion parameters?

The parameters in GPT-3, like any neural network, are the weights and biases of the layers. From the following table taken from the GTP-3 paper there are different versions of GPT-3 of various sizes. ...
  • 184
3 votes
Accepted

What tokenizer does OpenAI's GPT3 API use?

Tokenizer for GPT-3 is the same as GPT-2: https://huggingface.co/docs/transformers/model_doc/gpt2#gpt2tokenizerfast linked via: https://beta.openai.com/tokenizer
  • 146
3 votes
Accepted

How to access GPT-3, BERT or alike?

OpenAI has not released the weights of GPT-3, so you have to access it through their API. However, all other popular models have been released and are easily accessible. This includes GPT-2, BERT, ...
  • 17.9k
3 votes
Accepted

What is the difference between GPT blocks and BERT blocks

BERT is a Transformer encoder, while GPT is a Transformer decoder: You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the ...
  • 17.9k
3 votes
Accepted

For NLP, is GPT-3 better than RoBERTa?

They are meant for different purposes and they are hardly comparable. RoBERTa is meant for text classification and tagging tasks. The idea is that you take a pretrained RoBERTa model and finetune it ...
  • 17.9k
2 votes
Accepted

Does the transformer decoder reuse previous tokens' intermediate states like GPT2?

My understanding is that transformer decoders and transformer encoder-decoder models typically operate in the way that the GPT-2 does, i.e., representations in the generated sequence are computed once ...
2 votes

Evaluating Language Model on specific topic

What you can do is to compare against a validation set of the same domain. First, you use your LM to generate many sentences, and, for each sentence, you compute the BLEU score against the whole ...
  • 17.9k
2 votes

How Exactly Does In-Context Few-Shot Learning Actually Work in Theory (Under the Hood), Despite only Having a "Few" Support Examples to "Train On"?

I highly recommend you read Microsoft's recent paper about In Context Learning. Although the focus is on LLM I think it can be generalised to other models. The idea is to consider models as mesa|meta-...
2 votes
Accepted

Loss on whole sequences in Causal Language Model

The figure and the blog post are simply incorrect. Doing a reverse image search, I see that the image you posted comes from a blog post on Towards Data Science. That image is so wrong. Just think that ...
  • 17.9k
2 votes
Accepted

How to summarize a long text using GPT-3

Is there already a popular open-source script to do that? The Python library GPT Index (MIT license) can summarize a large document or collection of documents with GPT-3. From the documentation: <...
2 votes

In ChatGPT, what is the difference between Reinforcement-Learning-from-Human-Feedback and Data-Re-Label?

ChatGPT, being a generative model, generates sequences of tokens. There are no labels in the sense of a classification problem. Therefore, re-labeling using the reward signal does not make sense in ...
  • 17.9k
1 vote

ChatGPT's Architecture - Decoder Only? Or Encoder-Decoder?

Summary ChatGPT is the fine-tuning of GPT-3.5, which is a language model based on a Transformer decoder with some modifications with respect to the original Transformer architecture. Therefore it is a ...
  • 17.9k
1 vote

Which of these 2 approaches is the best route to learn to build a question answer chatbot?

I would choose option 2 (Build an application(chatbot) on top of existing algorithms like GPT-3 or BERT) for your particular use case, specifically GPT. For your case, it seems that you are trying to ...
1 vote

Empirical indications regarding demanded skills and tasks of data science jobs?

Quite general question, how you will divide your time working as a data scientist depends mostly on the job position itself, your company, and your skills. Personally, I am not a fan of, as you say it,...
1 vote

How does compute required scale with number of model parameters?

Recent models using transformers like Lambda have less than 100 trillion parameters, and it answers much better than most humans (if not all, as it has a huge volume of knowledge). I mean that the ...
1 vote
Accepted

Training Objective of language model for GPT3

This may be best understood with a bit more of context from the article: A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether ...
  • 17.9k
1 vote

Best strategy for extracting specific structured data from unstructured sentences

The task is a specific case of NER (technically NER is a sequence labeling task, a special case of classification). I think you would have two main options: Apply a pre-trained NER model: most deal ...
  • 24.4k
1 vote
Accepted

Evaluating Language Model on specific topic

I think there are (at least) two parts to take into account in evaluating such a model: Whether the generated text correctly relate to the input topic Whether the generated text is grammatically and ...
  • 24.4k
1 vote

GPT-3 API Documentation?

No - GPT-3 API is not currently public. However once you get access, the documentation can be found at https://beta.openai.com/api-ref.
1 vote

Generate text using user-supplied keywords

Yes fine-tuning GPT2 could help you through this objective. But the only concern is regarding the size of input data you have. To get a better performing model, you must a have larger input set.
1 vote

Does the transformer decoder reuse previous tokens' intermediate states like GPT2?

The past token internal states are reused both in GPT-2 and any other Transformer decoder. For example, in fairseq's implementation of the transformer, these previous states are received in ...
  • 17.9k

Only top scored, non community-wiki answers of a minimum length are eligible