Questions tagged [transformer]
Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.
446
questions
0
votes
1
answer
17
views
Confused about Q and K in Attention Mechanism
The following equation computes the attention scores:
A = softmax(QK / d)
I think Q and K are interchangeable, but why is one called Query and the other called Key?
- 3
0
votes
0
answers
14
views
Prefix tuning in LLM uses learnable vectors to fine tune the model
I would like to implement a new architecture for Transformer.
Below description is my thought.
Prefix tuning in LLM uses learnable vectors to fine tune the model.
Is there a way to use the output ...
- 25
0
votes
0
answers
24
views
Assign layers and weights in BERT
I print the weight names and shape of the BERT transformer. Now, I want to assign the printed weight to the layers in the transformers architecture:
In the following, I can assign query, key and ...
- 101
0
votes
0
answers
4
views
Accuracy Drop in ViT with Patch Embedding: Investigating the Impact of Added Convolutional Layers
I'm currently working on incorporating a patch embedding layer into my Vision Transformer (ViT). I've defined this layer using four 2D convolutional and initialized it with a normal distribution. The ...
0
votes
0
answers
23
views
Fine-tuning Hugging Face’s Llama Model with Unlabelled Data from PDFs from niche domain
I’m unsure about the next steps. Specifically, I have the following questions:
How can I prepare my unlabelled data for the fine-tuning process?
What’s the best way to fine-tune the Llama model with ...
0
votes
0
answers
15
views
What happens when I set is_decoder to True in the bert API from huggingface?
Please help me understand the implications of initialising the bert model from huggingface with is_decoder parameter set to True
...
- 821
0
votes
1
answer
56
views
Open-Source Large Language Models (LLM): Your experience and recommendation
I’m looking for an open-source LLM for a new project. I want to use it for instructions and to fine-tune the model to a specific domain like legal and rights. Some LLMs are open-source, but they didn’...
- 101
0
votes
1
answer
19
views
What is the input to an encoder-decoder transformer in next word prediction task?
I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
- 103
1
vote
1
answer
51
views
How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder
Bert has two types of tasks that it uses to learn contextual word embeddings:
Masked word prediction
Next sentence prediction
I have read the paper and even there the training details are a little ...
- 821
0
votes
0
answers
24
views
0
votes
0
answers
41
views
Can't overfit Transformer Encoder
In the below code I am trying to train a very simple Transformer Encoder model to basically do nothing with its input. Giving some arbitrary input vector x, the aim of the model is then to output that ...
0
votes
1
answer
31
views
Converting a Standard LSTM RNN over to a Transformer Model
I am looking for some advice on converting my existing CNN/LSTM RNN over to a Transformer type model. This regression model takes a sliding window size of 240 rows with 33 features. It aims to ...
- 111
0
votes
0
answers
54
views
Are there other pre-trained time series transformers than TimeGPT?
I am looking for pre-trained transformers trained on time-series data to use for transfer learning (for forecasting).
I found TimeGPT, and it does claim in this paper to be the first foundation model. ...
- 131
0
votes
0
answers
10
views
For a fine-tuning a transformer to type like a specific person, should I use sentence semantic embeddings or word semantic embeddings
I'm not clear on the pros and cons of each one for this particular task. Is there even a meaningful difference?
My guess is using semantic embeddings for words will be better in nearly all cases ...
0
votes
1
answer
37
views
Understanding Multi-headed Attention from architecture details
I've a conceptual question
BERT-base has a dimension of 768 for query, key and value and 12 heads (Hidden dimension=768, number of heads=12). The same is conveyed if we see the BERT-base architecture
<...
1
vote
0
answers
37
views
In rotary positional embeddings (RoPE), why do we not rotate the values as well?
Actually, the question is all there is
As per the paper I see that the rotations are applied only to the keys and the queries. Why are the rotations not applied to the values as well?
The reasons for ...
- 821
0
votes
1
answer
62
views
What is the "Extract" token and how is the final Linear layer applied in GPT?
In the manuscript of GPT, the authors have given the following image:
Questions:
What is the final "Extract" (token?)? Is it the "END" token?
How is the final linear layer ...
- 821
0
votes
0
answers
9
views
What is the prior mu in Heterogeneous Graph Transformer?
I am reading https://arxiv.org/pdf/2003.01332.pdf and do not understand what the prior (\mu) is supposed to be. I also found their implementation on github, but it is still not clear to me. For ...
- 101
1
vote
1
answer
60
views
A question about contextual embeddings in the decoder only transformer architecture (gpt)
I am reading up on the decoder only architecture
Relevant excerpts:
We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers):
$$\phi : V^L \to R^{d \times ...
- 821
0
votes
0
answers
33
views
RetNet Paper Multi Scale Retention dimemsion question
From the paper:
https://arxiv.org/pdf/2307.08621.pdf
But since X is of size n by $d_{model}$.
How can we compute $XW_Q$? Since the row length of X which is $d_{model}$ is not the same as the column ...
1
vote
0
answers
182
views
How to implement 2d Rotary Position Embedding in PyTorch?
The original RoPE paper suggests that the Rotary Position Embedding it describes can easily be extended to two or more dimensions: 3.2.2 in https://arxiv.org/abs/2104.09864 . I'm trying to find a ...
- 111
0
votes
0
answers
18
views
Why are 1/n, 2/n, 3/n ... 2048/n not good positional encodings to be concatenated to the word vectors in transformers?
The transformer architecture has no sense of the relative positions of the word and hence we need to pass that information apriori to the along with the word embeddings to the model
The positional ...
- 821
2
votes
3
answers
64
views
How can we use a transfomer model with new data if we still don't have the output?
Transformer models are trained using inputs and outputs. They are both embedded and encoded and used to train multi-head attention mechanisms...
But how can we use a transformer model to predict new ...
- 185
0
votes
0
answers
9
views
Deep Learning Methods for Video Classification
I'm working on a dataset with ~300 videos that last from 9 to 13-minute interviews of each subject and it has all the personality-related metadata that was collected during initial surveys. Which Deep ...
1
vote
1
answer
27
views
A good set of datasets/models for testing an NLP technique
I am a machine learning researcher who up until this point has primarily worked on Computer Vision problems. However, I have an idea for an NLP technique involving a novel Transformer architecture, ...
- 13
1
vote
1
answer
57
views
Why is T5 often used in text-to-data for text prompt encoders?
In the text-to-data(music, image, audio, etc.) generative AI field, one method of encoding text prompts is to use pre-trained language models. Such an approach was used in research on Moûsai [1] and ...
- 13
0
votes
1
answer
37
views
Neural regression predictions all around the mean of target
I have a transformer regression model and some data about last users transactions (categorical and numerical). My target has exponential distribution with mean aroud 10e4 and also zero-inflated, so I ...
0
votes
0
answers
215
views
How to get Llama-2 Rotary Embeddings?
I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output:
In the picture I highlight the rotary embeddings.
How can get the ...
- 101
0
votes
0
answers
17
views
Errors in results after saving model vs using directly from memory
I am trying to save a Fine Tuned model using trainer.save_model() but after I load the saved_model it just responds with the input back again and does not give any ...
- 191
0
votes
0
answers
45
views
Different generated patches from original image using vision transformer (ViT)
I am using ViT for image classification, I scaled images in range of [-1,1], and I also padded images. Then, I used the following code to see the original image and generated patches, but the output ...
- 1
0
votes
1
answer
42
views
Some fundamental questions about Transformer
In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e.,
\begin{align}
y^t = x^...
- 101
2
votes
0
answers
47
views
CLIP Visual Transformer image encoder
I was doing some experiments with the CLIP's visual transformer encoder output (clip-ViT-B-32). So basically given the same scene or image, it should output almost ...
- 217
0
votes
0
answers
44
views
Swin Transformer Relative Position Biases
I was reading the swin transformer paper and looking at the github implementation, i noticed that when calculating the relative position bias the input to the log function before the CPB MLP is scaled ...
- 1
0
votes
1
answer
34
views
Tensorflow diagram for attention mechanism
I was reading the tutorial from tensorflow on the transformer model, however, when they explain the transformer model, they display such a picture :
which I don't understand. What do the ingoing ...
- 101
1
vote
0
answers
47
views
Deploying a model with GPU and pay-per-inference
I may have the wrong stack exchange. If that's the case, could someone point me to a stack that could help with this. Anyways...
My backend employs a sentence transformer model from HuggingFace. Since ...
- 141
0
votes
0
answers
61
views
Sum of vector sentence embeddings vs. paragraph embedding
I have been experimenting with the all-MiniLM-L6-v2 model for computing 384-dimensional vector embeddings for text paragraphs. The following code compares the embedding computed for a paragraph with ...
- 141
0
votes
0
answers
17
views
How can BERT/Transformer models accept input batches of different sizes?
I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.
How is that possible? I thought we ...
- 101
1
vote
2
answers
515
views
how do we adapt LLM token embeddings with custom vocab
Hi im just getting started with understanding transformer based models and I am not able to find how the token embeddings are arrived at?. there are multiple tokenization approaches and multiple ...
- 111
0
votes
0
answers
26
views
Representation of 1D time series for use in transformers
I am working on a transformer where the encoder is given a sentence. However, the decoder is given 1D time series data (so, a single 1xm row vector). Given that the encoder is given n 350 dimensional ...
- 131
0
votes
0
answers
16
views
Differences between the two ViT-based models: GLP and DPT?
I've been learning to do some depth estimation tasks so I came across the GLP model (https://arxiv.org/pdf/2201.07436.pdf) which is suggested on huggingface website. I'm new to machine learning, so ...
- 101
2
votes
2
answers
2k
views
Understanding alpha parameter tuning in LORA paper
I was reading the LORA paper https://arxiv.org/pdf/2106.09685.pdf a thing I don’t understand is section 4.1, where the updates are updated by alpha, where alpha is a constant in r. It is said that ...
- 21
0
votes
1
answer
91
views
Do transformers output probabilities depend only on previous tokens?
In a transformer, the target sequence shifted right is placed in the input of the transformer decoder, and a upper triangular mask is provided so that the attention layer will not depend "tokens ...
- 3
0
votes
0
answers
633
views
Unsupervised fine tuning of Code LLMs
How to prepare code data to fine tune a code LLM in an unsupervised way or is it even possible?
For example:
Task: Code summarization with custom code base (with no summaries)
Let's assume that this ...
- 1
2
votes
2
answers
288
views
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models.
In the original "Attention is All You Need&...
0
votes
0
answers
289
views
Adapting a BERT-based model from HuggingFace for NER (named entity recognition) and RE (relation extraction)?
Context: NER (named entity recognition) and RE (relation extraction) from sentences obtained from radiology reports (medical text).
There is a BERT-based model from HuggingFace I would like to use for ...
- 377
0
votes
0
answers
25
views
Using ESM2 to predict GRAB sensor performance
I am an undergrad working in a neuroscience lab, and my PI has tasked me with using ESM2(https://github.com/facebookresearch/esm) to predict GPCR Activation Based (GRAB) sensor performance for in vivo ...
0
votes
0
answers
22
views
What Can Prevent Time-Series Prediction Model From Learning Trend?
I am building an encoder-decoder prediction model based on this paper:
https://www.sciencedirect.com/science/article/pii/S0952197623001483
It is made of a transformer encoder and a 1D CNN Decoder. The ...
- 1
0
votes
0
answers
34
views
Are there other "interactive" non-linear neural network layers besides self-attention layer?
In the self-attention layer
$$
\operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V
$$
$Q$, $K$ and $V$ are all linear with respect to embedding vectors $x$, ...
- 215
2
votes
3
answers
241
views
What does it mean order of input sequence does not matter for transformer self-attention head?
The need for positional encoding in transformer models is justified by permutation invariance of self-attention heads, because, without it, transformer wouldn't have any mechanism to take into account ...
- 215
0
votes
1
answer
33
views
How quickly can a transformer self-heal if you wipe out one of its layers?
Say we have a fully-trained N-layer transformer model (encoder-only, decoder-only, or encoder-decoder), with embedding dimension ...
- 977