Questions tagged [language-model]
Language models are used extensively in Natural Language Processing (NLP) and are probability distributions over a sequence of words or terms.
164
questions
0
votes
0
answers
21
views
Using LLMs for structured data?
I've been trying to work with structured data in language models, and it's proving to be quite challenging. I'm confident that with Langchain, I should be able to solve the problem, but I'm not ...
- 21
0
votes
0
answers
24
views
Assign layers and weights in BERT
I print the weight names and shape of the BERT transformer. Now, I want to assign the printed weight to the layers in the transformers architecture:
In the following, I can assign query, key and ...
- 101
0
votes
1
answer
43
views
Purely extractive Language Model
Given an email thread, I am trying to extract the body of the most recent email.
I used to do that with rules. Now I am testing Large Language Models (LLM) to see if I they provide a less ad hoc ...
- 103
0
votes
1
answer
59
views
Open-Source Large Language Models (LLM): Your experience and recommendation
I’m looking for an open-source LLM for a new project. I want to use it for instructions and to fine-tune the model to a specific domain like legal and rights. Some LLMs are open-source, but they didn’...
- 101
0
votes
1
answer
20
views
What is the input to an encoder-decoder transformer in next word prediction task?
I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
- 103
0
votes
0
answers
51
views
Why is 0.7, in general, the default value of temperature for LLMs?
I have recently read through a lot of documentation and articles about Large Language Models (LLMs), and I have come to the conclusion that 0.7 is, most of the time, the default value for the ...
- 1
0
votes
0
answers
7
views
Systematic way of selecting internet texts for a machine translation corpus / dataset?
I am currently working on a neural machine translation project and want to gather a corpus (or dataset) of internet texts that are written in standard and plain language. In theory, it certainly makes ...
- 101
0
votes
0
answers
9
views
【NLP】Is there a model or task that determines contextual similarity?
I am trying to work on an engagement detection task in which I have to determine if a student is engaged in class.
I am looking for an NLP approach where I can calculate the similarity score of a ...
- 1
0
votes
0
answers
19
views
Since LLMs only provide output in text, how can LLMs do 'action' such as search, run code etc?
Since LLMs only provide output in text, I wonder how LLMs can do 'action' such as search, run code etc. I try to delve deep into how langchain agent works and what I got is that the used prompt is ...
0
votes
0
answers
16
views
TFRobertaSequenceClassification for Address Normalization task
I have dataset with two column: one with faulty addresses, and other with correct addresses. I want to train a model such that, I can use it later for correcting all the incoming faulty addresses.
I ...
0
votes
0
answers
19
views
Harford 2023 model: explicit reference
I'm citing in a paper the Wizard-Vicuna Uncensored 30B model of Hartford (2023). But I don't have an exact bib reference other than various web links for that model. Could anyone help?
- 1
0
votes
0
answers
18
views
Semantic Scoring and readability for short sentences
I am working on short sentences for NLP based classification. I wish to make a assessment if a sentence is readable before training the system on it. Now readability scores are not working since ...
0
votes
0
answers
10
views
Using OSRM with Langchain
I am trying to develop a chatbot using Langchain, to give me a routing option between 2 points on a custom map data. I am using free APIs from Huggingface for embedding and Ollama model for the ...
- 1
0
votes
0
answers
230
views
How to read CSV File into Vector Store
I have a CSV file, and I am using langchain to read it into the vector store FAISS. My question is, since I have a CSV file, is RecursiveTextSplitter required? Put differently, consider the following ...
0
votes
0
answers
55
views
Decreasing the summary length from langchain load_summarize_chain?
How can i reduce the output size of the summarization in langchain map reduce method ?
0
votes
1
answer
17
views
Locating base.py when working on Colab
I have faced an error while working with langchain on colab. There is a post on github which recommends changing some configurations in
...
- 103
0
votes
1
answer
42
views
Why do we want to maximize the average log probability in neural language models?
I am currently trying to understand the Paragraph Vector framework by reading the paper "Distributed Representation of Sentences and Documents" by Quoc Le and Thomas Mikolov but I have ...
- 1
0
votes
0
answers
219
views
How to get Llama-2 Rotary Embeddings?
I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output:
In the picture I highlight the rotary embeddings.
How can get the ...
- 101
0
votes
0
answers
51
views
Specifying arguments of HuggingFaceHub
In this tutorial, when specifying
llm = HuggingFaceHub(
repo_id=repo_id, model_kwargs={"temperature": 0.5, "max_length": 64})
only ...
0
votes
1
answer
100
views
What is source_column argument in csv loader?
In this tutorial, what is the purpose of source_column argument? Does it act like a primary key in Databases? Thanks in advance.
...
0
votes
0
answers
37
views
Can LLM fine-tuning be used to improve a language?
I'm Danish, and with all the excitement around open LLM models, I'm feeling a little left out.
Take Llama 2, for example - it was trained on a very small Danish dataset. Just enough to learn the words ...
- 101
1
vote
1
answer
44
views
Why is Spacy sentiment score 0.0 for a sentence?
I'm trying to get a sentence's sentiment score using Spacy and apparently every sentence I pass gets a score of 0.0. Can someone help me understand what's going wrong here?
...
- 17
0
votes
0
answers
27
views
"text" parameter in pinecone call from langchain
In this tutorial, I do not understand what "text" refers to
vectorstore = Pinecone(index, embeddings.embed_query, "text")
Could you please help?...
-3
votes
1
answer
77
views
coversational AI chatbot using langchain and chatgpt 3.5?
can we develop end to end hotel booking coversational AI chatbot using langchain and chatgpt 3.5 ?
2
votes
1
answer
100
views
Implementation of spBLEU
I was looking for a way to explore evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does anyone have a ...
- 33
0
votes
0
answers
10
views
Create samples out of documents for Causal Language Modelling
I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
- 11
4
votes
2
answers
4k
views
Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?
It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving ...
- 143
0
votes
0
answers
119
views
Disadvantages of using just Vector Stores
I've just started with langchain, and one thing baffles me.
For starters, I'm using an in-memory vector store created from a local csv file
...
- 101
0
votes
1
answer
33
views
How quickly can a transformer self-heal if you wipe out one of its layers?
Say we have a fully-trained N-layer transformer model (encoder-only, decoder-only, or encoder-decoder), with embedding dimension ...
- 977
0
votes
0
answers
117
views
Fine-Tune Llama on main and auxiliary task
I am trying to fine-tune Llama model on two task at the same time, using hugging face library:
Main task: Causal language model like the model was initially trained for
A classification task based on ...
- 11
0
votes
0
answers
16
views
Error in seq2seq translation when passing predicted output to rnn due to input shape not always being the same
I'm working on a language translator and I'm getting an error I'm unsure about.
During the decoding process when using argmax on the predicted output I am sometimes getting an RuntimeError ...
0
votes
0
answers
47
views
What is the best LLM that can be used on a single GPU?
I am interested in the best/state-of-the-art Large Language model that can be used in a single GPU.
I read that Falcon 7B is state-of-the-art. Is there anything better? Any data that show the pros/...
- 101
2
votes
1
answer
42
views
Smart Selection of Training Data for Fine-tuning Language Models in Small Domains
Background
I am working to make language models (for example, Stanford's Alpaca model) perform well on a new small domain through fine-tuning on domain-specific dataset $D$.
If the size of $D$ is $N$, ...
- 83
0
votes
1
answer
1k
views
memory and context in LLM models
I have a large document and I may need to introduce a large part of it to my llm for insight generation I know that that text can be chunked into parts and with the right prompt I can get what I want ...
- 137
0
votes
0
answers
33
views
Language model gradient sensitivity to insignificant tokens
I'm trying out a method to identify important training samples for a given test-time prediction. What it essentially boils down to is calculating the gradient of a test-time prediction and ordering ...
- 11
3
votes
1
answer
1k
views
LMM Fine Tuning - Supervised Fine Tuning Trainer (SFTTrainer) vs transformers Trainer
When should one opt for the Supervised Fine Tuning Trainer (SFTTrainer) instead of the regular Transformers Trainer when it comes to instruction fine-tuning for Language Models (LLMs)? From what I ...
5
votes
3
answers
2k
views
Further Training a pre-trained LLM
My goal is to use the general knowledge and language understanding of a pre-trained LLM and to continue training on a smaller domain specific corpus to improve the model's knowledge on the domain. ...
- 81
1
vote
0
answers
737
views
Fine-tuning a pre-trained LLM for question-answering
Objective
My goal is to fine-tune a pre-trained LLM on a dataset about Manchester United's (MU's) 2021/22 season (they had a poor season). I want to be able to prompt the fine-tuned model with ...
- 13
1
vote
1
answer
97
views
How can models like Mosaic's MPT-7b or Bloombergs BLOOMGPT take in so many tokens?
I've read the paper on ALiBi, and I understand that these models are biasing the values made in the query/key multiplication.
But from my understanding, when I build the actual model I give it ...
- 113
1
vote
1
answer
229
views
LLM powered chat bot enhanced by NER
I have been reading on the capabilities of LLM based conversational agents and have been wondering if there is even possibility for any further enhancement with the addition of NER to such system.
If ...
0
votes
1
answer
135
views
Can we train the Dolly v-2 model on a large general purpose unlabelled text?
I am familiar with ML and Deep Learning concepts and have had a look at Dolly and even got the pretrained model running on a Jupyter lab notebook on Databricks.
However when I take a look at their ...
- 101
0
votes
1
answer
69
views
Easy question on autoregressive LLM
For LLM decoder, how exactly is the K, Q, V for each decoding step?
Say my input prompt is "today is a" (good day).
At t= 0 (generation step 0):
K, Q, V are the projections of the sequence (&...
- 123
1
vote
1
answer
96
views
Were any LLMs trained on Google books?
An important limiting factor on the performance of large language models, is the amount of training text available. Of course, using e.g. the Gutenberg archive of public domain books is an obvious ...
- 139
0
votes
0
answers
145
views
How to make sure which masking arguments I need to provide for calling torch.nn.Transformer model?
The forward function of PyTorch's Transformer implementation torch.nn.Transformer have a number of masking arguments that are all optional :
...
- 392
0
votes
1
answer
220
views
Passing target text to gpt2 and T5 for fine tuning to learn text generation task
I have text with each line in following format:
<text-1> some text-1 <text-2> some text-2 <text-3> some text-3
I want fine tune model to learn ...
- 103
2
votes
1
answer
533
views
What is purpose of stacking N=6 blocks of encoder and decoder in transformer?
I was trying to understand transformer architecture from "Attention is all you need" paper.
What is purpose of stacking $N=6$ blocks of encoder and decoder? Does higher blocks represent ...
- 179
1
vote
1
answer
61
views
What does it exactly mean by "different representation subspaces" in transformer?
I was trying to understand transformer architecture from "Attention is all you need" paper.
The paper says:
Multi-head attention allows the model to jointly attend to information from ...
- 179
0
votes
1
answer
88
views
How K and V are extracted from encoder output in transformer?
I was trying to understand transformer architecture from "Attention is all you need" paper. The paper shows following transformer architecture:
How $K$ and $V$ is extracted from $512$ ...
- 179
1
vote
1
answer
259
views
Understanding dimensions of vectors at various places in transformer architecture
I was trying to understand transformer architecture from "Attention is all you need" paper. It says following regarding dimensions of different vectors:
The input consists of queries and ...
- 179
1
vote
2
answers
770
views
Fine-tuned MLM based RoBERTa not improving performance
We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM.
We took some sample data points (2M+) & fine-tuned ...
- 11