Questions tagged [gpt]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
18 views

predict next career suggestion

I have a dataset having job and description. i want to make model which can predict what are the thing that user needs to improve when the user inputs his skills. For an example, If he has skills - ...
pycoder's user avatar
1 vote
0 answers
37 views

How can I use Time-GPT for pretraining my model

I am mentioning Time-GPT here as a placeholder example. It can be any pretrained model. Suppose I have a dataset that requires some time series prediction. How can I leverage a well-trained model and ...
Mohammad Mosiur's user avatar
0 votes
0 answers
12 views

Create a gpt-3.5 API request that determines whether any time range in a list intersects with a given time range

I've created a prompt that should select a requested number of employees from the list. But the step 1 doesn't work properly. Sometimes GPT takes in account only the time range and ignores the date. I ...
pazukdev's user avatar
  • 101
0 votes
0 answers
14 views

How are GPT2 token embedding vectors processed internally?

I am experimenting with the GPT2-XL model and trying to understand the internal structure. While I understand most of the components and how they affect the size of the activation tensors (such as ...
John Doe's user avatar
1 vote
1 answer
37 views

Falcon-7B llm giving random output

I am using a falcon 7B model for a chatbot without any finetuning with the following code ...
Saket Vempaty's user avatar
0 votes
1 answer
56 views

Do we really need a very large dataset to train GPTs?

Do we really need a very large dataset to train GPTs? If this dataset is not big, won't GPT work well? Or will it still work better than conventional learning models in this situation? And is it ...
nisar's user avatar
  • 1
0 votes
0 answers
27 views

data verctorizing of a free text field to get better responses of the prompt

wonder if best practices are existing to approach the solution "what kind of vectorizing of a input filed" is the best for my use-case. Model, size and others can have a significant impact ...
eric's user avatar
  • 1
0 votes
1 answer
37 views

Reinforcement learning

I am working on a sentiment analysis project. I used BERT model for training but lack of data it gives huge overfitting. So after i moved LLM approach to do that. Using LLM finally i got good results....
Sandun Tharaka's user avatar
0 votes
1 answer
107 views

What is the "Extract" token and how is the final Linear layer applied in GPT?

In the manuscript of GPT, the authors have given the following image: Questions: What is the final "Extract" (token?)? Is it the "END" token? How is the final linear layer ...
figs_and_nuts's user avatar
0 votes
0 answers
32 views

Is openAI text generation models an extension of embedding models?

we can creating embeddings using below code ...
Vinay Sharma's user avatar
1 vote
1 answer
146 views

A question about contextual embeddings in the decoder only transformer architecture (gpt)

I am reading up on the decoder only architecture Relevant excerpts: We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers): $$\phi : V^L \to R^{d \times ...
figs_and_nuts's user avatar
2 votes
1 answer
291 views

what is the difference between window size and context length of language model?

is window size and context length of language model one and the same thing? ******** following text is added as question with ONLY above text was not allowed ***** I am trying to understand how GPT ...
Vinay Sharma's user avatar
1 vote
0 answers
122 views

How does supervised fine-tuning work in InstructGPT?

See Figure 2 from the InstructGPT paper: I want to know how Step 1 works. Here is one possible algorithm. Pass the prompt through the model, and compute the negative log of the probability of the ...
jskattt797's user avatar
0 votes
1 answer
326 views

Why do GPT models use a transpose of the embedding matrix to convert outputs to logits?

According to section 3.1 of the original GPT paper, GPT right-multiplies the final output vectors (after applying a Transformer decoder model) by the transpose of the embedding matrix, before applying ...
jskattt797's user avatar
5 votes
2 answers
6k views

Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?

It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving ...
Ethan's user avatar
  • 153
1 vote
1 answer
422 views

How to measure accuracy of GPT model

I am working on a model to build questions automatically from some text My model will analyse provided article and ask authors questions that can help improving their articles How can we measure the ...
asmgx's user avatar
  • 539
0 votes
0 answers
145 views

Disadvantages of using just Vector Stores

I've just started with langchain, and one thing baffles me. For starters, I'm using an in-memory vector store created from a local csv file ...
dejanualex's user avatar
0 votes
0 answers
45 views

What is the input and output of GPT model for fine-tuning?

From my understanding, for the pretraining of GPT model, we need to do next token prediction task. In this case, Input -> The GPT models are general-purpose language models that can perform ... (...
Kyuwan's user avatar
  • 1
2 votes
1 answer
112 views

How was the token library constructed for ChatGPT / other GPT systems?

I have found literally hundreds of articles on Google with titles like 'What are tokens and how to use them,' but haven't been able to find any information at all on how the token libraries themselves ...
Sinnombre's user avatar
  • 153
0 votes
0 answers
29 views

Are there any studies on the actual loss of quality in a quantized GPT model? (eg GPTQ)

I've been trying to think of how different a GPT would perform if we swapped from floats to 8-bit. Given that we take say the top 10 words, how much fidelity is actually lost? How many tokens are ...
JuanIsFree's user avatar
2 votes
2 answers
372 views

Training GPT Model with Swagger Documents, Need Help with Model Fit

We are currently developing an application that performs actions based on user input using 'gpt-3.5-turbo-0613'. For example, if a user enters the prompt "create a user with the name Potter and ...
mostwelcome's user avatar
1 vote
1 answer
77 views

Why is the target output in GPT (Andrej Karpathy's course) contains copies of tokens from the input?

The question is based on Andrej Karpathy lecture on training a toy GPT model on Tiny Shakespeare dataset (youtube link). In this model, tokens are single characters with a dictionary of around 60 ...
DeLorean88's user avatar
0 votes
0 answers
518 views

GPT4All with Langchain generating gibberish in RHEL 8

I was trying to create a pipeline using Langchain and GPT4All (gpt4all-converted.bin). The pipeline ran fine when we tried on a windows system. But now when I am trying to run the same code on a RHEL ...
Saikat Bhattacharya's user avatar
0 votes
0 answers
25 views

Huggingface not saving model checkpoint

I am trying to train T5 model. This is how my training arguments look like: ...
RajS's user avatar
  • 103
1 vote
0 answers
698 views

Creating LLM chatbot using llama-index + langchain

As the title suggests: I'm trying to build a chatbot which his goal should be sort of like "chatgpt". The chatbot will be installed on Slack workspace and I'm struggling with which scope I ...
Ben's user avatar
  • 209
0 votes
0 answers
81 views

Question about GPT-2 causal masking for downstream tuning

I'm currently finetuning a pre-trained GPT-2 model for image captioning. Using a ViT for image encoding, the output embeddings are then fed into the GPT-2 model (from HuggingFace) as ...
Jeff Tian's user avatar
0 votes
1 answer
232 views

Passing target text to gpt2 and T5 for fine tuning to learn text generation task

I have text with each line in following format: <text-1> some text-1 <text-2> some text-2 <text-3> some text-3 I want fine tune model to learn ...
RajS's user avatar
  • 103
0 votes
0 answers
212 views

Fine tune Flan T5 to build a SlackBot

I want to create a Slack chatbot that will be able to "somehow" mock a "ChatGPT like" behavior. I want the bot to be able: answer questions understand from context summarize info ...
Ben's user avatar
  • 209
2 votes
1 answer
453 views

dolly v2 - how does it internally learn to follow instructions

this is more a curiosity query than anything else. The git repo for dolly gives us an easy way to swap, the training dataset , to train custom models, as long as we follow the format. I however have ...
Vikram Murthy's user avatar
0 votes
1 answer
19 views

Where can I see a summary of tools and techniques for most updated transformer developments?

Since the invention of chatGPT, there are many tools and techniques and variants invented ever since. I want to keep track of these developments and tools but I find no avail. May I know which ...
Student's user avatar
  • 411
1 vote
1 answer
4k views

Chromadb in my local system

I am trying to install chromadb on my local system but getting following error even after installing Microsoft C++ Build Tools: Is it because "Chromadb" can only be used in google Collab and ...
Spandan Bhatnagar's user avatar
3 votes
1 answer
226 views

Use a "ChatGPT like" engine really locally without exposing sensitive data

I was recently playing around with llama_index and llama Hub and found it very easy to be used on my own data set. While this is ...
Ben's user avatar
  • 209
12 votes
1 answer
12k views

How does an LLM "parameter" relate to a "weight" in a neural network?

I keep reading about how the latest and greatest LLMs have billions of parameters. As someone who is more familiar with standard neural nets but is trying to better understand LLMs, I'm curious if a ...
slim_wizard's user avatar
1 vote
0 answers
19 views

In ChatGPT, The difference of using reward to guide policy vs using the dataset of reward to train policy?

In ChatGPT, What are the differences of using reward to guide policy vs using the dataset of reward to train policy?
惊天补扣's user avatar
5 votes
5 answers
1k views

Is using GPT-4 to label data advisable?

If I have a lot of text data that needs to be labeled (e.g. sentiment analysis), and given the high accuracy of GPT-4, could I use it to label data? Or would that introduce bias or some other issues?
cookiecutter's user avatar
1 vote
1 answer
90 views

What should the numerical values of the <startofsentence> and <endofsentence> token vectors be?

I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I ...
Austin Capobianco's user avatar
0 votes
1 answer
381 views

What are the benefits of changing the temperature parameter in the chatgpt api?

(Cross-posted from: https://ai.stackexchange.com/q/39813/57463 after a week of no satisfactory answer) TLDR: What are the benefits of increasing the temperature in the chatGPT API ? Context (can be ...
userrandrand's user avatar
2 votes
1 answer
2k views

Can I use LLM to explain codebase?

I am a Data Engineer, and I am currently assigned a task to refactor an outdated code and rectify any bugs present. However, I am unable to comprehend the code written in the existing codebase. ...
Yiffany's user avatar
  • 158
3 votes
1 answer
939 views

How do GPT models go from token probabilities to textual outputs?

Suppose GPT-2 or GPT-3 is trying to generate the next token, and it has a probability distribution (after applying softmax to some output logits) for the different possible next tokens. How does it ...
jskattt797's user avatar
0 votes
0 answers
56 views

fine tuning open ai model with historical data

i'm trying to understand more about training models and unsure how to approach this problem. I have a bunch of historical financial data that I would like open ai to use as additional context when ...
Evan Bashir's user avatar
0 votes
1 answer
100 views

Where can I find gpt-35-turbo in the Microsoft Pricing calculator?

Where can I find gpt-35-turbo in the Microsoft Pricing calculator? I don't see gpt-35-turbo in the model list: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/ ...
Franck Dernoncourt's user avatar
4 votes
2 answers
16k views

ChatGPT: How to use long texts in prompt?

I like the website chatpdf.com a lot. You can upload a PDF file and then discuss the textual content of the file with the file "itself". It uses ChatGPT. I would like to program something ...
meyer_mit_ai's user avatar
0 votes
1 answer
266 views

GPT-2 architecture question

I am currently working on a NLP model that compares two comments and determines which one would be more popular. I have already came up with an architecture - it will be based on GPT-2. But now I am ...
Euler-Maskerony's user avatar
0 votes
0 answers
87 views

How to use LLMs to explain comman lines

I am interested to use LLMs to explain command lines. For POC, I used GPT-3 and got some really good results: ...
Alex L's user avatar
  • 101
0 votes
1 answer
784 views

What are the advantages of autoregressive over seq2seq?

Why are recent dialog agents, such as ChatGPT, BlenderBot3, and Sparrow, based on the decoder architecture instead of the encoder-decoder architecture? I know the difference between the attention of ...
Fei Wang's user avatar
0 votes
1 answer
546 views

Words limit using GPT-3 API and fine tune model

In the documentation for GPT-3 API, it says One limitation to keep in mind is that, for most models, a single API request can only process up to 2,048 tokens (roughly 1,500 words) between your prompt ...
Salty Gold Fish's user avatar
2 votes
1 answer
223 views

Does GPT-3 remember data from prompts used to fine tune it?

I am trying to fine tune a model using OpenAI's fine tuning API. I am passing bodies of text (for example, news paper articles) as prompts and the data I want from it as completions. Let us consider ...
Kushal Mohnot's user avatar
3 votes
1 answer
2k views

Does OpenAI and ChatGPT use Scikit Learn?

Scikit Learn is Python's go-to open source package for running common AI and machine learning algorithms. Does OpenAI and its product ChatGPT use or rely on Scikit Learn on its back-end at all? If not,...
user610620's user avatar
1 vote
1 answer
127 views

Some answers given by ChatGPT are just beyond ridiculous, what could be the reasons?

Some answers given by ChatGPT are just beyond ridiculous, especially in Chinese (I am Chinese so I ask ChatGPT in both Chinese and English). Here is an example, The last answer ...
Qiulang's user avatar
  • 113
3 votes
1 answer
158 views

Which Publicly Accessible Large Language Models are Very Similar to OpenAI's ChatGPT?

What other large language models exist or will soon exist that are VERY similar to OpenAI's ChatGPT in the sense of being fine-tuned or otherwise specifically created for conversational tasks ...
user141493's user avatar