9
votes
Accepted
Does fine-tuning require retraining the entire model?
No, you don't need to retrain the entire model. Fine-tuning refers to taking the weights trained in the general model and then continuing training a bit using your specific data. Using this approach, ...
- 537
7
votes
Accepted
How is GPT able to handle large vocabularies?
GPT-2 does not use a word-level vocabulary but a subword-level vocabulary, specifically byte-pair encoding (BPE). This means that it does not predict the next word but the next subword token.
BPE ...
- 17.9k
6
votes
What's the right input for gpt-2 in NLP
Updated answer
After reading @Jessica's answer, I carefully read the original GPT-2 paper and I confirm that the authors do not add special tokens, but simply the text ...
- 17.9k
6
votes
Does BERT has any advantage over GPT3?
This article on Medium introduces GPT-3 makes some comparisons with BERT.
Specifically, section 4 examines how GPT-3 and BERT differ and mentions that: "On the Architecture dimension, BERT still ...
- 161
5
votes
Does fine-tuning require retraining the entire model?
Yes. If open-sourced, we will be able to customize the model to our requirements. This is one of the most important modelling techniques called Transfer Learning
A pre-trained model, such as GPT-3, ...
- 625
4
votes
Does BERT has any advantage over GPT3?
BERT needs to be fine-tuned to do what you want.
GPT-3 cannot be fine-tuned (even if you had access to the actual weights, fine-tuning it would be very expensive)
If you have enough data for fine-...
- 141
4
votes
Accepted
Can I fine tune GPT-3?
The weights of GPT-3 are not public. You can fine-tune it but only through the interface provided by OpenAI. In any case, GPT-3 is too large to be trained on CPU.
About other similar models, like GPT-...
- 17.9k
4
votes
Accepted
How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>
BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained ...
- 17.9k
3
votes
BERT vs GPT architectural, conceptual and implemetational differences
To start with your last question: you correctly say that BERT is an encoder-only model trained with the masked language-modeling objective and operates non-autoregressively. GPT-2 is a decode-only ...
- 1,501
3
votes
What exactly are the parameters in GPT-3's 175 billion parameters?
The parameters in GPT-3, like any neural network, are the weights and biases of the layers.
From the following table taken from the GTP-3 paper
there are different versions of GPT-3 of various sizes. ...
- 184
3
votes
Accepted
What tokenizer does OpenAI's GPT3 API use?
Tokenizer for GPT-3 is the same as GPT-2:
https://huggingface.co/docs/transformers/model_doc/gpt2#gpt2tokenizerfast
linked via:
https://beta.openai.com/tokenizer
- 146
3
votes
Accepted
How to access GPT-3, BERT or alike?
OpenAI has not released the weights of GPT-3, so you have to access it through their API. However, all other popular models have been released and are easily accessible. This includes GPT-2, BERT, ...
- 17.9k
3
votes
Accepted
What is the difference between GPT blocks and BERT blocks
BERT is a Transformer encoder, while GPT is a Transformer decoder:
You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the ...
- 17.9k
3
votes
Accepted
For NLP, is GPT-3 better than RoBERTa?
They are meant for different purposes and they are hardly comparable.
RoBERTa is meant for text classification and tagging tasks. The idea is that you take a pretrained RoBERTa model and finetune it ...
- 17.9k
2
votes
Accepted
Does the transformer decoder reuse previous tokens' intermediate states like GPT2?
My understanding is that transformer decoders and transformer encoder-decoder models typically operate in the way that the GPT-2 does, i.e., representations in the generated sequence are computed once ...
- 36
2
votes
Evaluating Language Model on specific topic
What you can do is to compare against a validation set of the same domain. First, you use your LM to generate many sentences, and, for each sentence, you compute the BLEU score against the whole ...
- 17.9k
2
votes
How Exactly Does In-Context Few-Shot Learning Actually Work in Theory (Under the Hood), Despite only Having a "Few" Support Examples to "Train On"?
I highly recommend you read Microsoft's recent paper about In Context Learning. Although the focus is on LLM I think it can be generalised to other models.
The idea is to consider models as mesa|meta-...
- 41
2
votes
Accepted
Loss on whole sequences in Causal Language Model
The figure and the blog post are simply incorrect. Doing a reverse image search, I see that the image you posted comes from a blog post on Towards Data Science. That image is so wrong. Just think that ...
- 17.9k
2
votes
Accepted
How to summarize a long text using GPT-3
Is there already a popular open-source script to do that?
The Python library GPT Index (MIT license) can summarize a large document or collection of documents with GPT-3.
From the documentation:
<...
- 5,430
2
votes
In ChatGPT, what is the difference between Reinforcement-Learning-from-Human-Feedback and Data-Re-Label?
ChatGPT, being a generative model, generates sequences of tokens. There are no labels in the sense of a classification problem. Therefore, re-labeling using the reward signal does not make sense in ...
- 17.9k
1
vote
ChatGPT's Architecture - Decoder Only? Or Encoder-Decoder?
Summary
ChatGPT is the fine-tuning of GPT-3.5, which is a language model based on a Transformer decoder with some modifications with respect to the original Transformer architecture. Therefore it is a ...
- 17.9k
1
vote
Which of these 2 approaches is the best route to learn to build a question answer chatbot?
I would choose option 2 (Build an application(chatbot) on top of existing algorithms like GPT-3 or BERT) for your particular use case, specifically GPT.
For your case, it seems that you are trying to ...
1
vote
Empirical indications regarding demanded skills and tasks of data science jobs?
Quite general question, how you will divide your time working as a data scientist depends mostly on the job position itself, your company, and your skills.
Personally, I am not a fan of, as you say it,...
- 116
1
vote
How does compute required scale with number of model parameters?
Recent models using transformers like Lambda have less than 100 trillion parameters, and it answers much better than most humans (if not all, as it has a huge volume of knowledge).
I mean that the ...
- 4,229
1
vote
Accepted
Training Objective of language model for GPT3
This may be best understood with a bit more of context from the article:
A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether
...
- 17.9k
1
vote
Best strategy for extracting specific structured data from unstructured sentences
The task is a specific case of NER (technically NER is a sequence labeling task, a special case of classification).
I think you would have two main options:
Apply a pre-trained NER model: most deal ...
- 24.4k
1
vote
Accepted
Evaluating Language Model on specific topic
I think there are (at least) two parts to take into account in evaluating such a model:
Whether the generated text correctly relate to the input topic
Whether the generated text is grammatically and ...
- 24.4k
1
vote
GPT-3 API Documentation?
No - GPT-3 API is not currently public.
However once you get access, the documentation can be found at https://beta.openai.com/api-ref.
- 18.8k
1
vote
Generate text using user-supplied keywords
Yes fine-tuning GPT2 could help you through this objective. But the only concern is regarding the size of input data you have. To get a better performing model, you must a have larger input set.
- 11
1
vote
Does the transformer decoder reuse previous tokens' intermediate states like GPT2?
The past token internal states are reused both in GPT-2 and any other Transformer decoder.
For example, in fairseq's implementation of the transformer, these previous states are received in ...
- 17.9k
Only top scored, non community-wiki answers of a minimum length are eligible