Questions tagged [openai-gpt]
The openai-gpt tag has no usage guidance.
35
questions
0
votes
0
answers
7
views
How can I detect whether a conversational agent should ask for more details?
Context
I am trying to build a conversational agent that asks questions. If a satisfactory answer has been provided, or probing further is likely futile, the agent should move to the next question. ...
- 101
0
votes
1
answer
30
views
Loss on whole sequences in Causal Language Model
I'd like to know, from an implementation point of view, when training a Causal Transformer such as GPT-2, if making prediction on whole sequence at once and computing the loss on the whole sequence is ...
- 117
0
votes
0
answers
6
views
(How can) I test my openai fine-tuned model against standard Question Answering benchmarks?
I would like to automate the evaluation of my model but I do not know how to. Open to other ideas than benchmarks. Also I may want to remove human subjectivity as much as possible.
- 23
1
vote
1
answer
19
views
How does compute required scale with number of model parameters?
GPT-3 has 175 billion parameters, required ~$3.114 * 10^{23}$ FLOPS, and took approximately one month to train on ~10k Tesla V100 GPUs. It seems commonly stated that the brain has the equivalent of ~...
0
votes
1
answer
10
views
How can I use my fine-tuned model through openai's API?
I have read the docs extensively and cannot find how to specify that I want to use my fine-tuned model and not one of their pretrained models for my completion job.
- 23
1
vote
0
answers
23
views
Guide to Natural language Prompt programming for few-shot learning of Pretrained Language Models
I'm currently working on a project with the goal of producing AI content in the space of a content generation like blog writing, Instagram caption generation etc. Found the in-context few-shot ...
- 11
0
votes
0
answers
7
views
Conversational model returns empty string after a while
I've been experinmenting with Huggingface models and I've set up a chatbot with DialoGPT. It works pretty well, but after a while it stops answering and just returns empty strings.
Before this it will ...
- 1,248
0
votes
0
answers
103
views
How do I use GPT-J for document data extraction?
I want to extract data from documents (native pdf's with English language) using GPT-J but without using it's API. I have searched all documentation regarding GPT-J but haven't came across anything ...
- 1
0
votes
1
answer
18
views
Dataset with Multiple Choice Questions for fine tuning
I hope it's allowed to ask here, but I am looking for a dataset (the format is not that important) that is similar to SQuAD, but it also contains false answers to the questions. I wanna use it to fine ...
1
vote
0
answers
10
views
Conceptually, how to deal with facts and time in GPT-3 and Language Models
When exploring text generation using various large language models, I frequently come across generated text which presents facts which are plain out wrong. I am not talking about fake news or bias, ...
6
votes
1
answer
1k
views
BERT vs GPT architectural, conceptual and implemetational differences
In the BERT paper, I learnt that BERT is encoder-only model, that is it involves only transformer encoder blocks.
In the GPT paper, I learnt that GPT is decoder-only model, that is it involves only ...
- 185
0
votes
1
answer
19
views
Is there a ubiquitous web crawler that can generate a good language-specific dataset for training a transformer?
It seems like a lot of noteworthy AI tools are being trained on datasets generated by web crawlers rather than human-edited, human-compiled corpora (Facebook Translate, GPT-3). In general, it sounds ...
- 111
0
votes
0
answers
12
views
What is a good few-shot classifier function?
Is there any pre-written library or function which can receive a few examples of data values being classified and then extend that to new data values received?
- 111
1
vote
0
answers
14
views
Is there an AI function that can provide the definition of a word or phase with reasonably good accuracy?
I would like to make use of a software function which can provide the definition of a word or phrase. These words and phrases are in the realm of common knowledge - objects like "DVD player",...
- 111
0
votes
2
answers
701
views
What exactly are the parameters in GPT-3's 175 billion parameters?
What exactly are the parameters in GPT-3's 175 billion parameters? Are these the words in text on which model is trained?
- 109
1
vote
1
answer
434
views
What tokenizer does OpenAI's GPT3 API use?
I'm building an application for the API, but I would like to be able to count the number of tokens my prompt will use, before I submit an API call. Currently I often submit prompts that yield a 'too-...
1
vote
0
answers
43
views
How to derive Evidence Lower Bound in the paper "Zero-Shot Text-to-Image Generation"?
Can someone share the derivation of Evidence Lower Bound in this paper ?
Zero-Shot Text-to-Image Generation
The overall procedure can be viewed as maximizing the
evidence lower bound (ELB) (Kingma &...
- 11
0
votes
1
answer
109
views
Training Objective of language model for GPT3
On page 34 of OpenAI's GPT-3, there is a sentence demonstrating the limitation of objective function:
Our current objective weights every token equally and lacks a notion of what is most important to ...
- 105
1
vote
0
answers
17
views
How to take the keywords from the given dataset to train GPT-2 based chatbot?
I am working with a dataset that contains Questions on various Events conducted by a college and the corresponding answers for the queries. I am using this dataset to train a GPT-2 355M model to ...
2
votes
1
answer
14
views
Best strategy for extracting specific structured data from unstructured sentences
Given a list of sentences like this:
4 to 5 hours over a period of 16 weeks
1st session: 2.0-2.5 hours & 2nd session: 1.5-2.0 hours
Approximately 5-6 visits over the course of 5 months. Visit 1, ...
- 121
0
votes
1
answer
275
views
How to access GPT-3, BERT or alike?
I am interested in accessing NLP models mentioned in scientific papers, to replicate some results and experiment.
But I only see waiting lists https://openai.com/blog/openai-api/ and licenses granted ...
- 155
1
vote
1
answer
1k
views
What is the difference between GPT blocks and BERT blocks
Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though.
If GPT uses Decoder only and BERT uses ...
- 113
0
votes
1
answer
2k
views
What's the right input for gpt-2 in NLP
I'm fine-tuning pre-trained gpt-2 for text summarization. The dataset contains 'text' and 'reference summary'. So my question is how to add special tokens to get the right input format. Currently I'm ...
- 21
2
votes
2
answers
61
views
Evaluating Language Model on specific topic
I have finetuned a pretrained Language Model(GPT-2) on a custom dataset of mine. I would like a way of evaluating the ability of my model to generate sentences of a specific predefined topic, given in ...
- 123
6
votes
2
answers
7k
views
Does BERT has any advantage over GPT3?
I have read a couple of documents that explain in detail about the greater edge that GPT-3(Generative Pre-trained Transformer-3) has over BERT(Bidirectional Encoder Representation from Transformers). ...
- 163
1
vote
1
answer
285
views
GPT-3 API Documentation?
Has documentation of the GPT-3 API been made public?
I would be interested in keeping myself up to speed on the API's capability.
- 123
-1
votes
1
answer
1k
views
For NLP, is GPT-3 better than RoBERTa? [closed]
I am learning deep learning and I want to get into NLP. I have done LSTM, and now I am learning about vectorisation and transformers. Can you please tell me, which algorithm is more effective and ...
5
votes
1
answer
933
views
How is GPT able to handle large vocabularies?
From what I understand, GPT and GPT-2 are trained to predict the $N^{th}$ word in a sentence given the previous $N-1$ words. When the vocabulary size is very large (100k+ words) how is it able to ...
- 419
2
votes
1
answer
58
views
Generate text using user-supplied keywords
I've got a use case where I need to generate sentences based on a set of user supplied keywords. Here is an example of what I need:
User input:
End-User: Data Scientists
Region: Middle East
...
- 141
2
votes
2
answers
918
views
Does the transformer decoder reuse previous tokens' intermediate states like GPT2?
I recently read Jay Alammar's blogpost about GPT-2 (http://jalammar.github.io/illustrated-gpt2/) which I found quite clear appart from one point :
He explains that the decoder of GPT-2 processes input ...
- 165
3
votes
2
answers
73
views
How to generate a sentence with exactly N words?
Thanks to GPT2 pretrained model now it is possible to generate meaningful sequence of words with or without prefix. However a sentence should end with a proper endings (.,!,?). I am just wondering how ...
- 31
1
vote
0
answers
49
views
Pretrained Models for Keyword-Based Text Generation
I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2).
An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
- 121
2
votes
1
answer
4k
views
How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>
As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode ...
- 133
3
votes
0
answers
181
views
Fine tune gpt2 via huggingface API for domain specific LM
i am using the script in the examples folder to fine-tune the LM for a bot meant to deal with insurance related queries. So if someone were to type "i am looking to modify my ..." , the autocomplete ...
- 188
2
votes
0
answers
195
views
Paragraph Generator using BERT or GPT
I am trying to generate similar sentences, called paragraph generation. For example, what is the name of the eldest brother of ram? - For these paragraphs can be - who is the oldest brother of ram? , ...
- 804