Questions tagged [language-model]

Language models are used extensively in Natural Language Processing (NLP) and are probability distributions over a sequence of words or terms.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

How is a causal language model correctly fine-tuned?

I want to fine-tune an SLM like Phi-2 through the huggingface API. I am in doubt how to achieve that, because I see two ways to do that and I am wondering which is the correct way. The task is just a ...
Marcel Braasch's user avatar
0 votes
0 answers
19 views

Converting relational database into vector database

Is there any open-source tools for converting relational database to vector database to be used in llm applications? Which steps can be taken in the conversion?
heyula's user avatar
  • 37
0 votes
0 answers
10 views

What is system prompt in fine tuning of GPT3.5 for natural language to sql queries

What exactly is a system prompt while finetuning GPT3.5 or a language model in general? How can I build system prompt for the task of converting natural language to SQL queries
heyula's user avatar
  • 37
0 votes
0 answers
12 views

what is the main difference between ROUGE and BLUE?

Both (ROUGE, BLUE) are useful to find the similarity between machine generated summary and reference summary. what is the main difference?
Tovlk's user avatar
  • 43
0 votes
1 answer
20 views

What languages llama2 supports?

Which languages llama2 supports? I looked at the docs and huggingface but I couldn't find a list. Just it says usage in other languages than English as out-of-scope.
heyula's user avatar
  • 37
0 votes
1 answer
35 views

How can I get the list of pretrained large language models?

Is there any place I can get the list of pre-trained large language models in a neat way? Despite the most common ones like gpt, BARD, llama2, which llm do you suggest that can be used for RAG and ...
heyula's user avatar
  • 37
0 votes
1 answer
34 views

How to check the license of a LLM for specific use?

How to check if a large language model has a license allowing to fine tune the model and then publish it publicly? How can I be sure that I can use and fine-tune a large language model without ...
heyula's user avatar
  • 37
0 votes
1 answer
20 views

How to choose ideal pretrained model for fine-tuning?

I started to work with LLMs lately and want to know how people choose their pre-trained models in their fine-tuning tasks? What is the criteria to choose the base model and which factors affect?
heyula's user avatar
  • 37
0 votes
0 answers
26 views

Can I fine tune MedPaLM model

Is it possible to fine-tune MedPaLM and MedPaLM2; Google's llms trained using PaLM specialized for medical domain. Can we fine-tune these models further to get more specialized models?
heyula's user avatar
  • 37
0 votes
1 answer
42 views

Is Machine Reading Comprehension (MRC) outdated?

I recently went through some litterature about knowledge-enhanced language models and found connections with the Machine Reading Comprehension (MRC) task. However, I couldn't find papers more recent ...
Barbara Gendron's user avatar
1 vote
1 answer
136 views

How can I leverage machine learning for log analysis?

I am new to data science and trying to find possibilities of using datascience in tasks. I have a set of logs which I want to convert to json. The logs are more or less of same format and I can write ...
SUNITA GUPTA's user avatar
0 votes
0 answers
63 views

Using LLMs for structured data?

I've been trying to work with structured data in language models, and it's proving to be quite challenging. I'm confident that with Langchain, I should be able to solve the problem, but I'm not ...
Cosapocha's user avatar
0 votes
0 answers
30 views

Assign layers and weights in BERT

I print the weight names and shape of the BERT transformer. Now, I want to assign the printed weight to the layers in the transformers architecture: In the following, I can assign query, key and ...
Christian01's user avatar
0 votes
1 answer
91 views

Purely extractive Language Model

Given an email thread, I am trying to extract the body of the most recent email. I used to do that with rules. Now I am testing Large Language Models (LLM) to see if I they provide a less ad hoc ...
mirix's user avatar
  • 103
0 votes
1 answer
151 views

Open-Source Large Language Models (LLM): Your experience and recommendation

I’m looking for an open-source LLM for a new project. I want to use it for instructions and to fine-tune the model to a specific domain like legal and rights. Some LLMs are open-source, but they didn’...
Christian01's user avatar
0 votes
1 answer
83 views

What is the input to an encoder-decoder transformer in next word prediction task?

I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
mehsheenman's user avatar
1 vote
1 answer
451 views

Why is 0.7, in general, the default value of temperature for LLMs?

I have recently read through a lot of documentation and articles about Large Language Models (LLMs), and I have come to the conclusion that 0.7 is, most of the time, the default value for the ...
jmpion's user avatar
  • 11
0 votes
0 answers
8 views

Systematic way of selecting internet texts for a machine translation corpus / dataset?

I am currently working on a neural machine translation project and want to gather a corpus (or dataset) of internet texts that are written in standard and plain language. In theory, it certainly makes ...
tschomacker's user avatar
0 votes
0 answers
13 views

【NLP】Is there a model or task that determines contextual similarity?

I am trying to work on an engagement detection task in which I have to determine if a student is engaged in class. I am looking for an NLP approach where I can calculate the similarity score of a ...
Leo's user avatar
  • 1
0 votes
0 answers
24 views

Since LLMs only provide output in text, how can LLMs do 'action' such as search, run code etc?

Since LLMs only provide output in text, I wonder how LLMs can do 'action' such as search, run code etc. I try to delve deep into how langchain agent works and what I got is that the used prompt is ...
Jarun Vantanavijarn's user avatar
0 votes
0 answers
25 views

TFRobertaSequenceClassification for Address Normalization task

I have dataset with two column: one with faulty addresses, and other with correct addresses. I want to train a model such that, I can use it later for correcting all the incoming faulty addresses. I ...
learner_account's user avatar
0 votes
0 answers
19 views

Harford 2023 model: explicit reference

I'm citing in a paper the Wizard-Vicuna Uncensored 30B model of Hartford (2023). But I don't have an exact bib reference other than various web links for that model. Could anyone help?
cel's user avatar
  • 1
0 votes
0 answers
21 views

Semantic Scoring and readability for short sentences

I am working on short sentences for NLP based classification. I wish to make a assessment if a sentence is readable before training the system on it. Now readability scores are not working since ...
Vinay Varahabhotla's user avatar
0 votes
0 answers
12 views

Using OSRM with Langchain

I am trying to develop a chatbot using Langchain, to give me a routing option between 2 points on a custom map data. I am using free APIs from Huggingface for embedding and Ollama model for the ...
sidewala's user avatar
0 votes
0 answers
694 views

How to read CSV File into Vector Store

I have a CSV file, and I am using langchain to read it into the vector store FAISS. My question is, since I have a CSV file, is RecursiveTextSplitter required? Put differently, consider the following ...
Karl 17302's user avatar
0 votes
0 answers
93 views

Decreasing the summary length from langchain load_summarize_chain?

How can i reduce the output size of the summarization in langchain map reduce method ?
Aadhil Imam's user avatar
0 votes
1 answer
20 views

Locating base.py when working on Colab

I have faced an error while working with langchain on colab. There is a post on github which recommends changing some configurations in ...
IwillLearn's user avatar
0 votes
1 answer
64 views

Why do we want to maximize the average log probability in neural language models?

I am currently trying to understand the Paragraph Vector framework by reading the paper "Distributed Representation of Sentences and Documents" by Quoc Le and Thomas Mikolov but I have ...
LeeKed's user avatar
  • 1
0 votes
0 answers
299 views

How to get Llama-2 Rotary Embeddings?

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the ...
Christian01's user avatar
0 votes
0 answers
184 views

Specifying arguments of HuggingFaceHub

In this tutorial, when specifying llm = HuggingFaceHub( repo_id=repo_id, model_kwargs={"temperature": 0.5, "max_length": 64}) only ...
Karl 17302's user avatar
0 votes
2 answers
370 views

What is source_column argument in csv loader?

In this tutorial, what is the purpose of source_column argument? Does it act like a primary key in Databases? Thanks in advance. ...
Karl 17302's user avatar
0 votes
0 answers
76 views

Can LLM fine-tuning be used to improve a language?

I'm Danish, and with all the excitement around open LLM models, I'm feeling a little left out. Take Llama 2, for example - it was trained on a very small Danish dataset. Just enough to learn the words ...
mindplay.dk's user avatar
1 vote
1 answer
110 views

Why is Spacy sentiment score 0.0 for a sentence?

I'm trying to get a sentence's sentiment score using Spacy and apparently every sentence I pass gets a score of 0.0. Can someone help me understand what's going wrong here? ...
Fardeen Khan's user avatar
0 votes
0 answers
31 views

"text" parameter in pinecone call from langchain

In this tutorial, I do not understand what "text" refers to vectorstore = Pinecone(index, embeddings.embed_query, "text") Could you please help?...
Karl 17302's user avatar
-3 votes
1 answer
95 views

coversational AI chatbot using langchain and chatgpt 3.5?

can we develop end to end hotel booking coversational AI chatbot using langchain and chatgpt 3.5 ?
Aadhil Imam's user avatar
2 votes
1 answer
274 views

Implementation of spBLEU

I was looking for a way to explore evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does anyone have a ...
Prithvi's user avatar
  • 33
0 votes
0 answers
14 views

Create samples out of documents for Causal Language Modelling

I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
Dimits's user avatar
  • 11
7 votes
3 answers
7k views

Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?

It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving ...
Ethan's user avatar
  • 173
0 votes
0 answers
156 views

Disadvantages of using just Vector Stores

I've just started with langchain, and one thing baffles me. For starters, I'm using an in-memory vector store created from a local csv file ...
dejanualex's user avatar
0 votes
1 answer
43 views

How quickly can a transformer self-heal if you wipe out one of its layers?

Say we have a fully-trained N-layer transformer model (encoder-only, decoder-only, or encoder-decoder), with embedding dimension ...
Darren Cook's user avatar
0 votes
0 answers
137 views

Fine-Tune Llama on main and auxiliary task

I am trying to fine-tune Llama model on two task at the same time, using hugging face library: Main task: Causal language model like the model was initially trained for A classification task based on ...
Dimits's user avatar
  • 11
0 votes
0 answers
20 views

Error in seq2seq translation when passing predicted output to rnn due to input shape not always being the same

I'm working on a language translator and I'm getting an error I'm unsure about. During the decoding process when using argmax on the predicted output I am sometimes getting an RuntimeError ...
bailey.bailey's user avatar
0 votes
0 answers
55 views

What is the best LLM that can be used on a single GPU?

I am interested in the best/state-of-the-art Large Language model that can be used in a single GPU. I read that Falcon 7B is state-of-the-art. Is there anything better? Any data that show the pros/...
Dion's user avatar
  • 101
2 votes
1 answer
48 views

Smart Selection of Training Data for Fine-tuning Language Models in Small Domains

Background I am working to make language models (for example, Stanford's Alpaca model) perform well on a new small domain through fine-tuning on domain-specific dataset $D$. If the size of $D$ is $N$, ...
Mr.Robot's user avatar
0 votes
1 answer
1k views

memory and context in LLM models

I have a large document and I may need to introduce a large part of it to my llm for insight generation I know that that text can be chunked into parts and with the right prompt I can get what I want ...
Mohamed Amine's user avatar
0 votes
0 answers
36 views

Language model gradient sensitivity to insignificant tokens

I'm trying out a method to identify important training samples for a given test-time prediction. What it essentially boils down to is calculating the gradient of a test-time prediction and ordering ...
rasgaard's user avatar
3 votes
1 answer
2k views

LMM Fine Tuning - Supervised Fine Tuning Trainer (SFTTrainer) vs transformers Trainer

When should one opt for the Supervised Fine Tuning Trainer (SFTTrainer) instead of the regular Transformers Trainer when it comes to instruction fine-tuning for Language Models (LLMs)? From what I ...
Marvin Martin's user avatar
6 votes
3 answers
4k views

Further Training a pre-trained LLM

My goal is to use the general knowledge and language understanding of a pre-trained LLM and to continue training on a smaller domain specific corpus to improve the model's knowledge on the domain. ...
Arthuro's user avatar
  • 91
2 votes
1 answer
929 views

Fine-tuning a pre-trained LLM for question-answering

Objective My goal is to fine-tune a pre-trained LLM on a dataset about Manchester United's (MU's) 2021/22 season (they had a poor season). I want to be able to prompt the fine-tuned model with ...
Tom Bomer's user avatar
1 vote
1 answer
100 views

How can models like Mosaic's MPT-7b or Bloombergs BLOOMGPT take in so many tokens?

I've read the paper on ALiBi, and I understand that these models are biasing the values made in the query/key multiplication. But from my understanding, when I build the actual model I give it ...
Travasaurus's user avatar