Questions tagged [language-model]
Language models are used extensively in Natural Language Processing (NLP) and are probability distributions over a sequence of words or terms.
117
questions
0
votes
1
answer
19
views
What does Codex take as tokens?
The typical default for neural networks in natural language processing has been to take words as tokens.
OpenAI Codex is based on GPT-3, but also deals with source code. For source code in general, ...
- 115
0
votes
0
answers
24
views
How to use LLMs to explain comman lines
I am interested to use LLMs to explain command lines.
For POC, I used GPT-3 and got some really good results:
...
- 101
0
votes
2
answers
44
views
Data Preparation for next word prediction
In most places, I have seen that when preparing the training data and label for next-word prediction from the corpus one uses a fixed window size say of length 4, and then scans the subsequences of ...
- 101
0
votes
0
answers
28
views
How to fine-tune a large language model for translation in a multi-dataset setting?
The Problem
We need to translate from language N to language C. If it helps, N is a natural ...
0
votes
0
answers
20
views
Vision Language Models Similar to DeepMind's Flamingo Available for Commercial Purposes?
Does anyone know of any Vision Language Models similar to DeepMind's Flamingo model that will take in a video clip a user provides the model along with a user-defined question in a prompt about what ...
- 141
1
vote
1
answer
77
views
Some answers given by ChatGPT are just beyond ridiculous, what could be the reasons?
Some answers given by ChatGPT are just beyond ridiculous, especially in Chinese (I am Chinese so I ask ChatGPT in both Chinese and English). Here is an example,
The last answer
...
- 111
1
vote
0
answers
71
views
Which Publicly Accessible Large Language Models are Very Similar to OpenAI's ChatGPT?
What other large language models exist or will soon exist that are VERY similar to OpenAI's ChatGPT in the sense of being fine-tuned or otherwise specifically created for conversational tasks ...
- 141
4
votes
0
answers
80
views
How Does the Reward Model in ChatGPT Calculate Losses?
Reading the InstructGPT paper(which seems to be what ChatGPT was built off of), I found this equation for the reward function.
However, I'm struggling to understand how this equation is used to ...
- 141
1
vote
1
answer
191
views
Specifics about ChatGPT's Architecture
Does anyone know of reliable sources that have written about the architecture of OpenAI's ChatGPT - specifically regarding the following?:
Number of hidden layers
Number of attention heads
...
- 141
2
votes
4
answers
1k
views
ChatGPT's Architecture - Decoder Only? Or Encoder-Decoder?
Does ChatGPT use an encoder-decoder architecture, or a decoder-only architecture? I have been coming across Medium and TowardsDataScience articles suggesting that it has an encoder-decoder ...
- 141
0
votes
1
answer
17
views
Predicting a next word from a sentence of a different lenght than seen in training
I am building a custom Decoder-only transformer model, which is being trained on the task of Next Word Prediction. The training procedure is analogous to that of chat GPT models - the input to the ...
- 1
0
votes
1
answer
123
views
Is there any differences between using text-davinci-003 with the Azure API vs. with the OpenAI API? [closed]
I wonder whether there are any differences between using text-davinci-003 with the Azure API vs. with the OpenAI API.
- 5,478
0
votes
0
answers
13
views
FinBERT out of the box performance testing
I'm trying to perform an out of the box performance test for FinBERT using the financialphrasebank dataset(sentiment analysis) to get a baseline performance before I start finetuning the model. The ...
- 1
0
votes
0
answers
60
views
ChatGPT with multilingual language
How it can answer with multilingual langauge ?
Maybe it's because it use GPT-3.5, since their dataset has Common Crawl dataset, which has more than 40+ languages.
And as I found that InstructGPT's ...
- 11
1
vote
0
answers
28
views
Seeking Example CLIP Model Code Allowing Me to Use a Pretrained Model and Go Directly to Inference Without any Additional Training
I have been endlessly searching for open-source code (including from OpenAI themselves) which would allow me to take a pretrained CLIP model image encoder (e.g., the ViT B/32) and then solely based on ...
- 141
2
votes
1
answer
112
views
What size language model can you train on a GPU with x GB of memory?
I'm trying to figure out what size language model I will be able to train on a GPU with a certain amount of memory. Let's for simplicity say that 1 GB = 109 bytes; that means that, for example, on a ...
- 131
0
votes
0
answers
65
views
How to find the optimal number of samples for fine-tuning a pre-trained language model for text classification?
I'm trying to fine-tune a pre-trained language model (PLM) for text classification. The dataset that I'm using for fine-tuning includes about 40k samples.
I wonder if I should use the whole dataset or ...
- 115
-1
votes
1
answer
82
views
What are MLM and NSP models actually used for after they've been trained?
I am a Python programmer working with deep learning nets and I have recently built my own language models as well as I have fine-tuned popular models like BERT. MY question is - after these models ...
- 3
1
vote
1
answer
39
views
Sentiment Analysis models trained on articles / alternative data
For a 6 class sentence classification task (emotion), I have a list of sentences where I retrieve the sentiment using a language model that was trained on Tweets (bertweet).
It works fine for ...
- 195
7
votes
1
answer
489
views
How Exactly Does In-Context Few-Shot Learning Actually Work in Theory (Under the Hood), Despite only Having a "Few" Support Examples to "Train On"?
Recent models like the GPT-3 Language Model (Brown et al., 2020) and the Flamingo Visual-Language Model (Alayrac et al., 2022) use in-context few-shot learning. The models are able to make highly ...
- 141
1
vote
0
answers
17
views
Recommendations of NLP for classifying sentance into tense forms
I have a dataset of tweets. I have to classify each tweet into it's tense forms like whether it's about past, present or future. So for that can you please recommend any pretrained NLP model or method ...
5
votes
1
answer
65
views
Do large pretrained language models already "know" about NLP tasks?
Nowadays the state-of-the-art in NLP is to finetune a large pretrained language model such as BERT/GPT etc. on specfic tasks. These language models are pretrained on a huge amount of data and then ...
- 323
2
votes
0
answers
17
views
What explains T5's recent resurgence?
I read on https://towardsdatascience.com/choosing-the-right-language-model-for-your-nlp-use-case-1288ef3c4929:
I find the curve for T5 to be particularly interesting. What explains its recent ...
- 5,478
0
votes
0
answers
160
views
How can I run full BLOOM-176B on my laptop?
I am reading about the possibility to run BLOOM-1.3B on laptop using the simple Jupyter environment https://towardsdatascience.com/getting-started-with-bloom-9e3295459b65
My question is - is this ...
- 131
1
vote
0
answers
149
views
How are Learned Latent Arrays for the Perceiver Resampler in DeepMind's Flamingo Vision-Language Model Actually Calculated? By which Technique?
In "Flamingo: a Visual Language Model for Few-Shot Learning" (Alayrac et al. 2022) https://arxiv.org/abs/2204.14198 DeepMind makes use of "learned latent queries" in their "...
- 141
0
votes
0
answers
36
views
What does logits in Casual Language Modeling represent?
I am reading the docs for transformers by hugging face and I see that the logits produced by casual language models are of the shape (batch_size, sequence_length, config.vocab_size).
I also read the ...
0
votes
1
answer
14
views
Model for binary classification of links as "Article" or "Other"
I'm creating a web crawler which must:
Fetch a web page.
Parse all <a> tags with hrefs on the page.
Classify the tags as either article (Meaning the link ...
0
votes
1
answer
27
views
Do I need training data in multiple languages for a multilingual transformer?
I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.
...
- 189
1
vote
1
answer
321
views
Loss on whole sequences in Causal Language Model
I'd like to know, from an implementation point of view, when training a Causal Transformer such as GPT-2, if making prediction on whole sequence at once and computing the loss on the whole sequence is ...
- 127
0
votes
0
answers
16
views
Arguments of OpenIE to extract fewer event triples
I'm new to NLP and I'm trying to using OpenIE to extract event triples from texts.
I looked into its documents but quite don't understand its arguments. For example, ...
- 165
0
votes
0
answers
20
views
Does the Lempel-Ziv-Welch algorithm have theoretical or practical use as a language model?
If we encode a string using the LZW algorithm, we obtain a dictionary which maps strings of increasing length onto output symbols and a sequence of output symbols.
Is the LZW algorithm useful (...
0
votes
0
answers
31
views
What is the minimal number of examples for BERT-like language model for the model to train a word
I have heard rumors of a particular count of positive examples that allowed the model to train a given word (or context of it - when talking about MLM) to be ~40. I am wondering though about the ...
- 209
1
vote
1
answer
17
views
Which format is preferrable to publish book dataset (plain or preprocessed)?
When I decide to publish collection of book texts as a dataset, should I do some preprocessing first or should I publish "plain texts"?
For example, https://huggingface.co/datasets/...
- 113
0
votes
0
answers
25
views
Clarification on "predict the next character given the previous 100 characters"
I am studying Justin Johnson's lecture on RNNs
Lecture recording: https://www.youtube.com/watch?v=dUzLD91Sj-o&list=PL5-TkQAfAZFbzxjBHtzdVCWE0Zbhomg7r&index=12&t=3177s
One of the examples ...
- 101
2
votes
1
answer
71
views
A multi label text classification problem
I'm looking to solve a multi label text classification problem but I don't really know how to formulate it correctly so I can look it up.. Here is my problem :
Say I have the document ...
- 35
3
votes
0
answers
42
views
Understanding Kneser-Ney Formula for implementation
I am trying to implement this formula in Python
$$ \frac{\text{max}(c_{KN}(w^{i}_{i-n+1} - d), 0)}{c_{KN}(w^{i-1}_{i-n+1})} + \lambda(c_{KN}(w^{i-1}_{i-n+1})\mathbb{P}(c_{KN}(w_{i}|w^{i-1}_{i-n+2})$$
...
- 237
0
votes
1
answer
60
views
Sequence-to-Sequence Transformer for Neural machine translation
I am using the tutorial in Keras documentation here. I am new to deep learning. On a different dataset Menyo-20k dataset, of about 10071 total pairs, 7051 training ...
- 17
1
vote
1
answer
135
views
Why not rule-based semantic role labelling?
I have recently found some interest in automatic semantic role labelling. Most introductory texts (e.g. Jurafsky and Martin, 2008) present approaches based on supervised machine learning, often using ...
- 113
1
vote
0
answers
14
views
Conceptually, how to deal with facts and time in GPT-3 and Language Models
When exploring text generation using various large language models, I frequently come across generated text which presents facts which are plain out wrong. I am not talking about fake news or bias, ...
1
vote
0
answers
125
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
0
votes
0
answers
149
views
What does the term "seed lexicon" means?
I am reading a research paper (NLP) and found the phrase "seed lexicon".
Could someone please explain it in detail?
Edit :
A sample paper
Leveraging Affective Bidirectional Transformers for ...
- 101
0
votes
1
answer
206
views
how to improve my imbalanced data NLP model?
I want to classify a patient's health as a prediction probability and get the top 10 most ill patients in a hospital. I have patient's condition notes, medical notes, diagnoses notes, and lab notes ...
- 138
2
votes
2
answers
164
views
Best approach for text classification of phrases with little syntactic difference
So I have the task of classifying sentences based on their level of 'change talk' shown. Change talk is a psychology term used in counseling sessions to express how much the client wants to change ...
- 73
2
votes
1
answer
33
views
Phrase/Token labeling
Looking for suggestions on how to define the following NLP problem and different ways in which it can be modeled to leverage machine learning. I believe there are multiple ways to model this problem. ...
- 21
1
vote
1
answer
135
views
For text classification, would a BoW or Word Embeddings based model ever be better than a Language Model?
I've done a bit of research, with this being the best as far as objectively measuring quality, but wanted to ask from a theoretical perspective if BoW-based models (e.g. using TF-IDF) or word ...
- 111
1
vote
0
answers
39
views
NLP-Problem, language model BERT?
Right now I am in the process of deciding on my masters thesis topic. Right now I and my professor are thinking about the possibility of having a large dataset of product requirements given in a ...
- 11
1
vote
0
answers
72
views
Why we shift target(output) by one offset in language modelling
I have been working in sequence prediction tasks (very similar to language modelling) where I want to predict the next token(s)/item(s) given past sequence of tokens. I have always taken an approach ...
- 316
0
votes
1
answer
28
views
Implementing a model for a language to another
I have a dataset of sentences of language X and Y (2 columns, for example, "abc def lang" ==> "xyz pqrt mno uages"). I want to have a output as a table that translates word by ...
- 11
0
votes
1
answer
310
views
How to predict the sentiment of the entities form the tweet?
I have a JSON file (tweets.json) that contains tweets (sentences) along with the name of
the author.
Objective 1: Get the most frequent entities from the tweets.
Objective 2: Find out the sentiment/...
- 129
1
vote
1
answer
72
views
Dealing with high frequency tokens during masked Language modelling?
Suppose I am working with a Masked Language Model to pre-train on a specific dataset. In that dataset, most sequences have a particular token of a high frequency
...
- 197