Questions tagged [openai-gpt]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
1answer
10 views

Best strategy for extracting specific structured data from unstructured sentences

Given a list of sentences like this: 4 to 5 hours over a period of 16 weeks 1st session: 2.0-2.5 hours & 2nd session: 1.5-2.0 hours Approximately 5-6 visits over the course of 5 months. Visit 1, ...
0
votes
1answer
33 views

How to access GPT-3, BERT or alike?

I am interested in accessing NLP models mentioned in scientific papers, to replicate some results and experiment. But I only see waiting lists https://openai.com/blog/openai-api/ and licenses granted ...
0
votes
1answer
71 views

What is the difference between GPT blocks and BERT blocks

Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though. If GPT uses Decoder only and BERT uses ...
-1
votes
1answer
141 views

What's the right input for gpt-2 in NLP

I'm fine-tuning pre-trained gpt-2 for text summarization. The dataset contains 'text' and 'reference summary'. So my question is how to add special tokens to get the right input format. Currently I'm ...
2
votes
2answers
54 views

Evaluating Language Model on specific topic

I have finetuned a pretrained Language Model(GPT-2) on a custom dataset of mine. I would like a way of evaluating the ability of my model to generate sentences of a specific predefined topic, given in ...
5
votes
2answers
1k views

Does BERT has any advantage over GPT3?

I have read a couple of documents that explain in detail about the greater edge that GPT-3(Generative Pre-trained Transformer-3) has over BERT(Bidirectional Encoder Representation from Transformers). ...
1
vote
1answer
114 views

GPT-3 API Documentation?

Has a documentation of the GPT-3 API been made public? I would be interested in keeping myself up to speed on the API's capability. Thanks!
0
votes
0answers
146 views

Does GPT-2 has pre trained for sentiment analysis?

I tried sentiment analysis with 345M model of GPT-2. But it took a long time to train. So is there any other GPT-2 model available for sentiment analysis?
-1
votes
1answer
324 views

For NLP, is GPT-3 better than RoBERTa? [closed]

I am learning deep learning and I want to get into NLP. I have done LSTM, and now I am learning about vectorisation and transformers. Can you please tell me, which algorithm is more effective and ...
3
votes
1answer
192 views

How is GPT able to handle large vocabularies?

From what I understand, GPT and GPT-2 are trained to predict the $N^{th}$ word in a sentence given the previous $N-1$ words. When the vocabulary size is very large (100k+ words) how is it able to ...
1
vote
1answer
28 views

Generate text using user-supplied keywords

I've got a use case where I need to generate sentences based on a set of user supplied keywords. Here is an example of what I need: User input: End-User: Data Scientists Region: Middle East ...
0
votes
0answers
14 views

How should we cut the long text into different sections for GPT-2 ? Do we use [PAD] to ensure the complete sentence included?

For GPT-2 model (I believe same for BERT) we need to cut the long text into fixed length for pre-training. Just wondering the details of how this should be implemented. Do we need to cut the text ...
1
vote
2answers
410 views

Does the transformer decoder reuse previous tokens' intermediate states like GPT2?

I recently read Jay Alammar's blogpost about GPT-2 (http://jalammar.github.io/illustrated-gpt2/) which I found quite clear appart from one point : He explains that the decoder of GPT-2 processes input ...
3
votes
2answers
41 views

How to generate a sentence with exactly N words?

Thanks to GPT2 pretrained model now it is possible to generate meaningful sequence of words with or without prefix. However a sentence should end with a proper endings (.,!,?). I am just wondering how ...
1
vote
0answers
32 views

Pretrained Models for Keyword-Based Text Generation

I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2). An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
1
vote
1answer
2k views

How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>

As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode ...
3
votes
0answers
113 views

Fine tune gpt2 via huggingface API for domain specific LM

i am using the script in the examples folder to fine-tune the LM for a bot meant to deal with insurance related queries. So if someone were to type "i am looking to modify my ..." , the autocomplete ...
2
votes
0answers
152 views

Paragraph Generator using BERT or GPT

I am trying to generate similar sentences, called paragraph generation. For example, what is the name of the eldest brother of ram? - For these paragraphs can be - who is the oldest brother of ram? , ...