19 votes

ChatGPT's Architecture - Decoder Only? Or Encoder-Decoder?

Summary ChatGPT is the fine-tuning of GPT-3.5, which is a language model based on a Transformer decoder with some modifications with respect to the original Transformer architecture. Therefore it is a ...
noe's user avatar
  • 26.6k
14 votes
Accepted

What tokenizer does OpenAI's GPT3 API use?

Tokenizer for GPT-3 is the same as GPT-2: https://huggingface.co/docs/transformers/model_doc/gpt2#gpt2tokenizerfast linked via: https://beta.openai.com/tokenizer UPDATE March 2023 For newer models, ...
pjama's user avatar
  • 256
14 votes
Accepted

How does an LLM "parameter" relate to a "weight" in a neural network?

Yes, the parameters in a large language model (LLM) are similar to the weights in a standard neural network. In both LLMs and neural networks, these parameters are numerical values that start as ...
pparker's user avatar
  • 392
11 votes
Accepted

How is GPT able to handle large vocabularies?

GPT-2 does not use a word-level vocabulary but a subword-level vocabulary, specifically byte-pair encoding (BPE). This means that it does not predict the next word but the next subword token. BPE ...
noe's user avatar
  • 26.6k
9 votes
Accepted

Does fine-tuning require retraining the entire model?

No, you don't need to retrain the entire model. Fine-tuning refers to taking the weights trained in the general model and then continuing training a bit using your specific data. Using this approach, ...
Andy's user avatar
  • 650
8 votes

What's the right input for gpt-2 in NLP

Updated answer After reading @Jessica's answer, I carefully read the original GPT-2 paper and I confirm that the authors do not add special tokens, but simply the text ...
noe's user avatar
  • 26.6k
8 votes
Accepted

Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?

There are many contributing factors to the abundance of research based on BERT vs the research based on Llama: Age: BERT has been around for far longer than Llama (2018 vs 2023), so it has more ...
noe's user avatar
  • 26.6k
7 votes

ChatGPT: How to use long texts in prompt?

I didn't find the site you mention that useful - perhaps it is not using the latest GPT model? Or ChatGPT does not yet have very good ability to understand the pdf file I provided it. Boring answer #1 ...
Valentas's user avatar
  • 1,209
6 votes

What exactly are the parameters in GPT-3's 175 billion parameters?

The parameters in GPT-3, like any neural network, are the weights and biases of the layers. From the following table taken from the GTP-3 paper there are different versions of GPT-3 of various sizes. ...
Derek H's user avatar
  • 214
6 votes
Accepted

What is the difference between GPT blocks and BERT blocks

BERT is a Transformer encoder, while GPT is a Transformer decoder: You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the ...
noe's user avatar
  • 26.6k
6 votes

Does BERT has any advantage over GPT3?

This article on Medium introduces GPT-3 makes some comparisons with BERT. Specifically, section 4 examines how GPT-3 and BERT differ and mentions that: "On the Architecture dimension, BERT still ...
Langley's user avatar
  • 161
6 votes

Does OpenAI and ChatGPT use Scikit Learn?

Based on the limited amount of code in OpenAI's GitHub, one of the primary packages is PyTorch. There is a much smaller amount of scikit-learn code. Since OpenAI has not released any code for ChatGPT, ...
Brian Spiering's user avatar
6 votes
Accepted

How do GPT models go from token probabilities to textual outputs?

Any language model can generate text with different approaches: Greedy decoding: you get the highest probability token at each time step. Sampling: the generated token is sampled from the probability ...
noe's user avatar
  • 26.6k
5 votes
Accepted

How to summarize a long text using GPT-3

Is there already a popular open-source script to do that? The Python library GPT Index (MIT license) can summarize a large document or collection of documents with GPT-3. From the documentation: <...
Franck Dernoncourt's user avatar
5 votes

Does fine-tuning require retraining the entire model?

Yes. If open-sourced, we will be able to customize the model to our requirements. This is one of the most important modelling techniques called Transfer Learning A pre-trained model, such as GPT-3, ...
user5722540's user avatar
5 votes

BERT vs GPT architectural, conceptual and implemetational differences

To start with your last question: you correctly say that BERT is an encoder-only model trained with the masked language-modeling objective and operates non-autoregressively. GPT-2 is a decode-only ...
Jindřich's user avatar
  • 1,741
4 votes
Accepted

How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>

BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained ...
noe's user avatar
  • 26.6k
4 votes

Does BERT has any advantage over GPT3?

BERT needs to be fine-tuned to do what you want. GPT-3 cannot be fine-tuned (even if you had access to the actual weights, fine-tuning it would be very expensive) If you have enough data for fine-...
MWB's user avatar
  • 141
4 votes
Accepted

Loss on whole sequences in Causal Language Model

The figure and the blog post are simply incorrect. Doing a reverse image search, I see that the image you posted comes from a blog post on Towards Data Science. That image is so wrong. Just think that ...
noe's user avatar
  • 26.6k
4 votes
Accepted

Can I fine tune GPT-3?

The weights of GPT-3 are not public. You can fine-tune it but only through the interface provided by OpenAI. In any case, GPT-3 is too large to be trained on CPU. About other similar models, like GPT-...
noe's user avatar
  • 26.6k
4 votes

Is using GPT-4 to label data advisable?

I agree with Jonathan Oren - in general, GPT-4 works fairly well for straightforward sentiment analysis, e.g. product reviews. One caveat though is that there are most certainly biases inherent in the ...
brewmaster321's user avatar
4 votes
Accepted

How was the token library constructed for ChatGPT / other GPT systems?

ChatGPT uses byte-pair encoding (BPE) as tokenization strategy. This approach was first proposed in the scientific article Neural Machine Translation of Rare Words with Subword Units. BPE uses subword-...
noe's user avatar
  • 26.6k
4 votes
Accepted

what is the difference between window size and context length of language model?

Yes, in large language models, window and context length refer to the same thing: the maximum token sequence length that the language model can handle at once.
noe's user avatar
  • 26.6k
4 votes
Accepted

How do I prompt GPT-4 to look at a PDF in Jupyter Notebook?

OpenAI's API does not support providing PDFs. Therefore, what you intend to do is not possible. You have other options though: Parse the PDF locally and send text to GPT4. The parse may consist of ...
noe's user avatar
  • 26.6k
3 votes

How Exactly Does In-Context Few-Shot Learning Actually Work in Theory (Under the Hood), Despite only Having a "Few" Support Examples to "Train On"?

I highly recommend you read Microsoft's recent paper about In Context Learning. Although the focus is on LLM I think it can be generalised to other models. The idea is to consider models as mesa|meta-...
Xmaster6y's user avatar
3 votes
Accepted

How to access GPT-3, BERT or alike?

OpenAI has not released the weights of GPT-3, so you have to access it through their API. However, all other popular models have been released and are easily accessible. This includes GPT-2, BERT, ...
noe's user avatar
  • 26.6k
3 votes
Accepted

For NLP, is GPT-3 better than RoBERTa?

They are meant for different purposes and they are hardly comparable. RoBERTa is meant for text classification and tagging tasks. The idea is that you take a pretrained RoBERTa model and finetune it ...
noe's user avatar
  • 26.6k
3 votes
Accepted

Does the transformer decoder reuse previous tokens' intermediate states like GPT2?

My understanding is that transformer decoders and transformer encoder-decoder models typically operate in the way that the GPT-2 does, i.e., representations in the generated sequence are computed once ...
Kevin Gimpel's user avatar
3 votes

Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?

Adding/complementing the other answers, BERT gives the possibility to access/obtain the embeddings of the fed input (which wasn't and still isn't the case of some other models). The embeddings are ...
ahmed_khan_89's user avatar
3 votes
Accepted

Why do GPT models use a transpose of the embedding matrix to convert outputs to logits?

As you described in your comment, this technique regularizes the model by reusing the same weights for the logit computation. I think, however, that there is previous reference to this technique prior ...
noe's user avatar
  • 26.6k

Only top scored, non community-wiki answers of a minimum length are eligible