Skip to main content

Questions tagged [language-model]

Language models are used extensively in Natural Language Processing (NLP) and are probability distributions over a sequence of words or terms.

Filter by
Sorted by
Tagged with
0 votes
1 answer
98 views

Why do we want to maximize the average log probability in neural language models?

I am currently trying to understand the Paragraph Vector framework by reading the paper "Distributed Representation of Sentences and Documents" by Quoc Le and Thomas Mikolov but I have ...
1 vote
1 answer
617 views

How are Learned Latent Arrays for the Perceiver Resampler in DeepMind's Flamingo Vision-Language Model Actually Calculated? By which Technique?

In "Flamingo: a Visual Language Model for Few-Shot Learning" (Alayrac et al. 2022) https://arxiv.org/abs/2204.14198 DeepMind makes use of "learned latent queries" in their "...
0 votes
2 answers
43 views

How to choose ideal pretrained model for fine-tuning?

I started to work with LLMs lately and want to know how people choose their pre-trained models in their fine-tuning tasks? What is the criteria to choose the base model and which factors affect?
0 votes
1 answer
162 views

Can we train the Dolly v-2 model on a large general purpose unlabelled text?

I am familiar with ML and Deep Learning concepts and have had a look at Dolly and even got the pretrained model running on a Jupyter lab notebook on Databricks. However when I take a look at their ...
1 vote
1 answer
386 views

How K and V are extracted from encoder output in transformer?

I was trying to understand transformer architecture from "Attention is all you need" paper. The paper shows following transformer architecture: How $K$ and $V$ is extracted from $512$ ...
0 votes
1 answer
125 views

Optimal input setup for character-level text classification RNN

I want to classify 500-character long text samples as to whether they look like natural language using a character-level RNN. I'm unsure as to the best way to feed the input to the RNN. Here are two ...
2 votes
3 answers
298 views

State-of-the-art Python packages that can evaluate language similarity

I am trying to evaluate the likelihood of generating a specific sentence out of a large set of sentences. To do this, I start from a simple approach: training a custom n-gram language model and ...
1 vote
2 answers
1k views

Fine-tuned MLM based RoBERTa not improving performance

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM. We took some sample data points (2M+) & fine-tuned ...
0 votes
1 answer
348 views

How to predict the sentiment of the entities form the tweet?

I have a JSON file (tweets.json) that contains tweets (sentences) along with the name of the author. Objective 1: Get the most frequent entities from the tweets. Objective 2: Find out the sentiment/...
4 votes
1 answer
563 views

In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens?

When learning Add-1 smoothing, I found that somehow we are adding 1 to each word in our vocabulary, but not considering start-of-sentence and end-of-sentence as two words in the vocabulary. Let me ...
1 vote
1 answer
144 views

Transfer learning between Language Model and classification

Following this fast.ai lecture, I am trying to understand the mechanism of Transfer Learning in NLP from a general Language Model (LM) to a classification problem. What is exactly taken from the ...
0 votes
0 answers
54 views

How much improvement does OpenAI o1 achieve from the chain of thought?

https://openai.com/index/learning-to-reason-with-llms/ OpenAI o1 also add more data than the last version of LLM.
1 vote
1 answer
2k views

Why is 0.7, in general, the default value of temperature for LLMs?

I have recently read through a lot of documentation and articles about Large Language Models (LLMs), and I have come to the conclusion that 0.7 is, most of the time, the default value for the ...
0 votes
1 answer
344 views

how to improve my imbalanced data NLP model?

I want to classify a patient's health as a prediction probability and get the top 10 most ill patients in a hospital. I have patient's condition notes, medical notes, diagnoses notes, and lab notes ...
0 votes
0 answers
47 views

For image+text, how is pre-training of Multimodal LLM generally done?

For image+text without video, how is pre-training of Multimodal Large Language Model generally done? Choice-1: Transform image to text, and then input all the text to LLM? Choice-2: Transform image to ...
0 votes
0 answers
27 views

Generating transaction data for a dataset to train on

My project is to predict what payment option a customer might use depending on various factors on a checkout screen. For example here are some of the fields I would have Variables : User_Location ...
18 votes
2 answers
34k views

Word2Vec embeddings with TF-IDF

When you train the word2vec model (using for instance, gensim) you supply a list of words/sentences. But there does not seem to be a way to specify weights for the words calculated for instance using ...
0 votes
0 answers
9 views

What are the key quality metrics for large language model releases?

I am a first year PhD student working on improving the release practices of Machine Learning Models, especially pre-trained large language models. I want to understand the above concept for a ...
0 votes
0 answers
11 views

What is query generation re-ranking method?

I am reading up on reranking methodologies that leverage LLMs. Relevant literature. One of the methods suggested is query generation Or, the same methodology from another source The task is to rank ...
0 votes
0 answers
19 views

How to find out that a conversation with a chatbot is likely ended

I'm working on a ChatBot with Python and langchain, and I'd like to have a metric that I could use to understand how close we ...
1 vote
1 answer
60 views

Callback handlers in Langchain

This might be an odd question, but why is there two codes for the class BaseCallbackHandler? https://api.python.langchain.com/en/latest/_modules/langchain_core/callbacks/base.html#BaseCallbackHandler ...
9 votes
3 answers
17k views

Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?

It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving ...
0 votes
0 answers
60 views

How is a causal language model correctly fine-tuned?

I want to fine-tune an SLM like Phi-2 through the huggingface API. I am in doubt how to achieve that, because I see two ways to do that and I am wondering which is the correct way. The task is just a ...
0 votes
0 answers
79 views

Converting relational database into vector database

Is there any open-source tools for converting relational database to vector database to be used in llm applications? Which steps can be taken in the conversion?
0 votes
0 answers
26 views

What is system prompt in fine tuning of GPT3.5 for natural language to sql queries

What exactly is a system prompt while finetuning GPT3.5 or a language model in general? How can I build system prompt for the task of converting natural language to SQL queries
0 votes
0 answers
21 views

what is the main difference between ROUGE and BLUE?

Both (ROUGE, BLUE) are useful to find the similarity between machine generated summary and reference summary. what is the main difference?
0 votes
1 answer
44 views

How can I get the list of pretrained large language models?

Is there any place I can get the list of pre-trained large language models in a neat way? Despite the most common ones like gpt, BARD, llama2, which llm do you suggest that can be used for RAG and ...
0 votes
1 answer
38 views

What languages llama2 supports?

Which languages llama2 supports? I looked at the docs and huggingface but I couldn't find a list. Just it says usage in other languages than English as out-of-scope.
0 votes
1 answer
62 views

How to check the license of a LLM for specific use?

How to check if a large language model has a license allowing to fine tune the model and then publish it publicly? How can I be sure that I can use and fine-tune a large language model without ...
0 votes
0 answers
92 views

Can I fine tune MedPaLM model

Is it possible to fine-tune MedPaLM and MedPaLM2; Google's llms trained using PaLM specialized for medical domain. Can we fine-tune these models further to get more specialized models?
3 votes
1 answer
2k views

Fine-tuning a pre-trained LLM for question-answering

Objective My goal is to fine-tune a pre-trained LLM on a dataset about Manchester United's (MU's) 2021/22 season (they had a poor season). I want to be able to prompt the fine-tuned model with ...
1 vote
2 answers
722 views

What is source_column argument in csv loader?

In this tutorial, what is the purpose of source_column argument? Does it act like a primary key in Databases? Thanks in advance. ...
0 votes
1 answer
43 views

Is Machine Reading Comprehension (MRC) outdated?

I recently went through some litterature about knowledge-enhanced language models and found connections with the Machine Reading Comprehension (MRC) task. However, I couldn't find papers more recent ...
17 votes
7 answers
23k views

ChatGPT's Architecture - Decoder Only? Or Encoder-Decoder?

Does ChatGPT use an encoder-decoder architecture, or a decoder-only architecture? I have been coming across Medium and TowardsDataScience articles suggesting that it has an encoder-decoder ...
1 vote
1 answer
536 views

How can I leverage machine learning for log analysis?

I am new to data science and trying to find possibilities of using datascience in tasks. I have a set of logs which I want to convert to json. The logs are more or less of same format and I can write ...
2 votes
1 answer
230 views

The differences between BNF and JSGF in NLP?

I wonder what the differences are between the BNF(Backus-Naur Form) and JSGF(Java Speech Grammar Format)? The former is a kind of context-free grammar taught in CS224, but I learned that the latter is ...
0 votes
1 answer
160 views

Purely extractive Language Model

Given an email thread, I am trying to extract the body of the most recent email. I used to do that with rules. Now I am testing Large Language Models (LLM) to see if I they provide a less ad hoc ...
0 votes
1 answer
203 views

Open-Source Large Language Models (LLM): Your experience and recommendation

I’m looking for an open-source LLM for a new project. I want to use it for instructions and to fine-tune the model to a specific domain like legal and rights. Some LLMs are open-source, but they didn’...
0 votes
1 answer
318 views

What is the input to an encoder-decoder transformer in next word prediction task?

I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
3 votes
1 answer
468 views

Were any LLMs trained on Google books?

An important limiting factor on the performance of large language models, is the amount of training text available. Of course, using e.g. the Gutenberg archive of public domain books is an obvious ...
1 vote
1 answer
217 views

Predicting a next word from a sentence of a different lenght than seen in training

I am building a custom Decoder-only transformer model, which is being trained on the task of Next Word Prediction. The training procedure is analogous to that of chat GPT models - the input to the ...
0 votes
0 answers
27 views

TFRobertaSequenceClassification for Address Normalization task

I have dataset with two column: one with faulty addresses, and other with correct addresses. I want to train a model such that, I can use it later for correcting all the incoming faulty addresses. I ...
3 votes
1 answer
185 views

Which Publicly Accessible Large Language Models are Very Similar to OpenAI's ChatGPT?

What other large language models exist or will soon exist that are VERY similar to OpenAI's ChatGPT in the sense of being fine-tuned or otherwise specifically created for conversational tasks ...
8 votes
1 answer
12k views

What size language model can you train on a GPU with x GB of memory?

I'm trying to figure out what size language model I will be able to train on a GPU with a certain amount of memory. Let's for simplicity say that 1 GB = 109 bytes; that means that, for example, on a ...
0 votes
0 answers
1k views

How to read CSV File into Vector Store

I have a CSV file, and I am using langchain to read it into the vector store FAISS. My question is, since I have a CSV file, is RecursiveTextSplitter required? Put differently, consider the following ...
0 votes
1 answer
22 views

Locating base.py when working on Colab

I have faced an error while working with langchain on colab. There is a post on github which recommends changing some configurations in ...
7 votes
3 answers
5k views

Further Training a pre-trained LLM

My goal is to use the general knowledge and language understanding of a pre-trained LLM and to continue training on a smaller domain specific corpus to improve the model's knowledge on the domain. ...
1 vote
0 answers
36 views

"text" parameter in pinecone call from langchain

In this tutorial, I do not understand what "text" refers to vectorstore = Pinecone(index, embeddings.embed_query, "text") Could you please help?...
1 vote
1 answer
287 views

Why is Spacy sentiment score 0.0 for a sentence?

I'm trying to get a sentence's sentiment score using Spacy and apparently every sentence I pass gets a score of 0.0. Can someone help me understand what's going wrong here? ...
-3 votes
1 answer
96 views

coversational AI chatbot using langchain and chatgpt 3.5?

can we develop end to end hotel booking coversational AI chatbot using langchain and chatgpt 3.5 ?