23
votes
ChatGPT's Architecture - Decoder Only? Or Encoder-Decoder?
Summary
ChatGPT is the fine-tuning of GPT-3.5, which is a language model based on a Transformer decoder with some modifications with respect to the original Transformer architecture. Therefore it is a ...
- 27.2k
15
votes
Accepted
How does an LLM "parameter" relate to a "weight" in a neural network?
Yes, the parameters in a large language model (LLM) are similar to the weights in a standard neural network. In both LLMs and neural networks, these parameters are numerical values that start as ...
- 422
14
votes
Accepted
What tokenizer does OpenAI's GPT3 API use?
Tokenizer for GPT-3 is the same as GPT-2:
https://huggingface.co/docs/transformers/model_doc/gpt2#gpt2tokenizerfast
linked via:
https://beta.openai.com/tokenizer
UPDATE March 2023
For newer models, ...
- 256
11
votes
Accepted
How is GPT able to handle large vocabularies?
GPT-2 does not use a word-level vocabulary but a subword-level vocabulary, specifically byte-pair encoding (BPE). This means that it does not predict the next word but the next subword token.
BPE ...
- 27.2k
10
votes
Accepted
Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?
There are many contributing factors to the abundance of research based on BERT vs the research based on Llama:
Age: BERT has been around for far longer than Llama (2018 vs 2023), so it has more ...
- 27.2k
9
votes
Accepted
Does fine-tuning require retraining the entire model?
No, you don't need to retrain the entire model. Fine-tuning refers to taking the weights trained in the general model and then continuing training a bit using your specific data. Using this approach, ...
- 650
8
votes
What's the right input for gpt-2 in NLP
Updated answer
After reading @Jessica's answer, I carefully read the original GPT-2 paper and I confirm that the authors do not add special tokens, but simply the text ...
- 27.2k
7
votes
What exactly are the parameters in GPT-3's 175 billion parameters?
The parameters in GPT-3, like any neural network, are the weights and biases of the layers.
From the following table taken from the GTP-3 paper
there are different versions of GPT-3 of various sizes. ...
- 224
7
votes
Accepted
What is the difference between GPT blocks and BERT blocks
BERT is a Transformer encoder, while GPT is a Transformer decoder:
You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the ...
- 27.2k
7
votes
ChatGPT: How to use long texts in prompt?
I didn't find the site you mention that useful - perhaps it is not using the latest GPT model? Or ChatGPT does not yet have very good ability to understand the pdf file I provided it.
Boring answer #1 ...
- 1,254
6
votes
Does BERT has any advantage over GPT3?
This article on Medium introduces GPT-3 makes some comparisons with BERT.
Specifically, section 4 examines how GPT-3 and BERT differ and mentions that: "On the Architecture dimension, BERT still ...
- 161
6
votes
Does OpenAI and ChatGPT use Scikit Learn?
Based on the limited amount of code in OpenAI's GitHub, one of the primary packages is PyTorch. There is a much smaller amount of scikit-learn code.
Since OpenAI has not released any code for ChatGPT, ...
- 22.1k
6
votes
Accepted
How do GPT models go from token probabilities to textual outputs?
Any language model can generate text with different approaches:
Greedy decoding: you get the highest probability token at each time step.
Sampling: the generated token is sampled from the probability ...
- 27.2k
5
votes
Does fine-tuning require retraining the entire model?
Yes. If open-sourced, we will be able to customize the model to our requirements. This is one of the most important modelling techniques called Transfer Learning
A pre-trained model, such as GPT-3, ...
- 675
5
votes
BERT vs GPT architectural, conceptual and implemetational differences
To start with your last question: you correctly say that BERT is an encoder-only model trained with the masked language-modeling objective and operates non-autoregressively. GPT-2 is a decode-only ...
- 1,771
5
votes
Accepted
How to summarize a long text using GPT-3
Is there already a popular open-source script to do that?
The Python library GPT Index (MIT license) can summarize a large document or collection of documents with GPT-3.
From the documentation:
<...
- 5,750
4
votes
Accepted
How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>
BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained ...
- 27.2k
4
votes
Does BERT has any advantage over GPT3?
BERT needs to be fine-tuned to do what you want.
GPT-3 cannot be fine-tuned (even if you had access to the actual weights, fine-tuning it would be very expensive)
If you have enough data for fine-...
- 141
4
votes
Accepted
Loss on whole sequences in Causal Language Model
The figure and the blog post are simply incorrect. Doing a reverse image search, I see that the image you posted comes from a blog post on Towards Data Science. That image is so wrong. Just think that ...
- 27.2k
4
votes
Accepted
Can I fine tune GPT-3?
The weights of GPT-3 are not public. You can fine-tune it but only through the interface provided by OpenAI. In any case, GPT-3 is too large to be trained on CPU.
About other similar models, like GPT-...
- 27.2k
4
votes
ChatGPT's Architecture - Decoder Only? Or Encoder-Decoder?
TLDR (simplified):
encoder sees into future, decoder predicts
transformer sees into future and then predicts, encodes, then decodes
gpt doesnt see into future, it only predicts - thats why it's ...
- 141
4
votes
Is using GPT-4 to label data advisable?
I agree with Jonathan Oren - in general, GPT-4 works fairly well for straightforward sentiment analysis, e.g. product reviews. One caveat though is that there are most certainly biases inherent in the ...
- 1,360
4
votes
Accepted
How was the token library constructed for ChatGPT / other GPT systems?
ChatGPT uses byte-pair encoding (BPE) as tokenization strategy. This approach was first proposed in the scientific article Neural Machine Translation of Rare Words with Subword Units. BPE uses subword-...
- 27.2k
4
votes
Accepted
what is the difference between window size and context length of language model?
Yes, in large language models, window and context length refer to the same thing: the maximum token sequence length that the language model can handle at once.
- 27.2k
4
votes
Accepted
How do I prompt GPT-4 to look at a PDF in Jupyter Notebook?
OpenAI's API does not support providing PDFs. Therefore, what you intend to do is not possible. You have other options though:
Parse the PDF locally and send text to GPT4. The parse may consist of ...
- 27.2k
3
votes
How Exactly Does In-Context Few-Shot Learning Actually Work in Theory (Under the Hood), Despite only Having a "Few" Support Examples to "Train On"?
I highly recommend you read Microsoft's recent paper about In Context Learning. Although the focus is on LLM I think it can be generalised to other models.
The idea is to consider models as mesa|meta-...
- 61
3
votes
Accepted
How to access GPT-3, BERT or alike?
OpenAI has not released the weights of GPT-3, so you have to access it through their API. However, all other popular models have been released and are easily accessible. This includes GPT-2, BERT, ...
- 27.2k
3
votes
Accepted
For NLP, is GPT-3 better than RoBERTa?
They are meant for different purposes and they are hardly comparable.
RoBERTa is meant for text classification and tagging tasks. The idea is that you take a pretrained RoBERTa model and finetune it ...
- 27.2k
3
votes
Accepted
Does the transformer decoder reuse previous tokens' intermediate states like GPT2?
My understanding is that transformer decoders and transformer encoder-decoder models typically operate in the way that the GPT-2 does, i.e., representations in the generated sequence are computed once ...
- 46
3
votes
Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?
Adding/complementing the other answers, BERT gives the possibility to access/obtain the embeddings of the fed input (which wasn't and still isn't the case of some other models).
The embeddings are ...
- 131
Only top scored, non community-wiki answers of a minimum length are eligible