Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Prefix tuning in LLM uses learnable vectors to fine tune the model

I would like to implement a new architecture for Transformer. Below description is my thought. Prefix tuning in LLM uses learnable vectors to fine tune the model. Is there a way to use the output ...
jackson's user avatar
  • 25
0 votes
0 answers
21 views

Assign layers and weights in BERT

I print the weight names and shape of the BERT transformer. Now, I want to assign the printed weight to the layers in the transformers architecture: In the following, I can assign query, key and ...
Christian01's user avatar
0 votes
0 answers
4 views

Accuracy Drop in ViT with Patch Embedding: Investigating the Impact of Added Convolutional Layers

I'm currently working on incorporating a patch embedding layer into my Vision Transformer (ViT). I've defined this layer using four 2D convolutional and initialized it with a normal distribution. The ...
External_Happy_77's user avatar
0 votes
0 answers
13 views

Fine-tuning Hugging Face’s Llama Model with Unlabelled Data from PDFs from niche domain

I’m unsure about the next steps. Specifically, I have the following questions: How can I prepare my unlabelled data for the fine-tuning process? What’s the best way to fine-tune the Llama model with ...
Divya Patel's user avatar
0 votes
0 answers
12 views

What happens when I set is_decoder to True in the bert API from huggingface?

Please help me understand the implications of initialising the bert model from huggingface with is_decoder parameter set to True ...
figs_and_nuts's user avatar
0 votes
1 answer
52 views

Open-Source Large Language Models (LLM): Your experience and recommendation

I’m looking for an open-source LLM for a new project. I want to use it for instructions and to fine-tune the model to a specific domain like legal and rights. Some LLMs are open-source, but they didn’...
Christian01's user avatar
0 votes
1 answer
19 views

What is the input to an encoder-decoder transformer in next word prediction task?

I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
mehsheenman's user avatar
1 vote
1 answer
47 views

How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder

Bert has two types of tasks that it uses to learn contextual word embeddings: Masked word prediction Next sentence prediction I have read the paper and even there the training details are a little ...
figs_and_nuts's user avatar
0 votes
0 answers
20 views

Pytorch Transformer only generating NaN when using mask

When I generate a src_mask like this ...
kot's user avatar
  • 1
0 votes
0 answers
37 views

Can't overfit Transformer Encoder

In the below code I am trying to train a very simple Transformer Encoder model to basically do nothing with its input. Giving some arbitrary input vector x, the aim of the model is then to output that ...
SeñorDavid's user avatar
0 votes
1 answer
28 views

Converting a Standard LSTM RNN over to a Transformer Model

I am looking for some advice on converting my existing CNN/LSTM RNN over to a Transformer type model. This regression model takes a sliding window size of 240 rows with 33 features. It aims to ...
Ted Wilmont's user avatar
0 votes
0 answers
48 views

Are there other pre-trained time series transformers than TimeGPT?

I am looking for pre-trained transformers trained on time-series data to use for transfer learning (for forecasting). I found TimeGPT, and it does claim in this paper to be the first foundation model. ...
Avatrin's user avatar
  • 131
0 votes
0 answers
10 views

For a fine-tuning a transformer to type like a specific person, should I use sentence semantic embeddings or word semantic embeddings

I'm not clear on the pros and cons of each one for this particular task. Is there even a meaningful difference? My guess is using semantic embeddings for words will be better in nearly all cases ...
Austin Capobianco's user avatar
0 votes
1 answer
37 views

Understanding Multi-headed Attention from architecture details

I've a conceptual question BERT-base has a dimension of 768 for query, key and value and 12 heads (Hidden dimension=768, number of heads=12). The same is conveyed if we see the BERT-base architecture <...
Namburi Srinath's user avatar
1 vote
0 answers
33 views

In rotary positional embeddings (RoPE), why do we not rotate the values as well?

Actually, the question is all there is As per the paper I see that the rotations are applied only to the keys and the queries. Why are the rotations not applied to the values as well? The reasons for ...
figs_and_nuts's user avatar
0 votes
1 answer
59 views

What is the "Extract" token and how is the final Linear layer applied in GPT?

In the manuscript of GPT, the authors have given the following image: Questions: What is the final "Extract" (token?)? Is it the "END" token? How is the final linear layer ...
figs_and_nuts's user avatar
0 votes
0 answers
9 views

What is the prior mu in Heterogeneous Graph Transformer?

I am reading https://arxiv.org/pdf/2003.01332.pdf and do not understand what the prior (\mu) is supposed to be. I also found their implementation on github, but it is still not clear to me. For ...
Servus's user avatar
  • 101
1 vote
1 answer
52 views

A question about contextual embeddings in the decoder only transformer architecture (gpt)

I am reading up on the decoder only architecture Relevant excerpts: We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers): $$\phi : V^L \to R^{d \times ...
figs_and_nuts's user avatar
0 votes
0 answers
32 views

RetNet Paper Multi Scale Retention dimemsion question

From the paper: https://arxiv.org/pdf/2307.08621.pdf But since X is of size n by $d_{model}$. How can we compute $XW_Q$? Since the row length of X which is $d_{model}$ is not the same as the column ...
KaizerBox's user avatar
1 vote
0 answers
156 views

How to implement 2d Rotary Position Embedding in PyTorch?

The original RoPE paper suggests that the Rotary Position Embedding it describes can easily be extended to two or more dimensions: 3.2.2 in https://arxiv.org/abs/2104.09864 . I'm trying to find a ...
LogicChains's user avatar
0 votes
0 answers
17 views

Why are 1/n, 2/n, 3/n ... 2048/n not good positional encodings to be concatenated to the word vectors in transformers?

The transformer architecture has no sense of the relative positions of the word and hence we need to pass that information apriori to the along with the word embeddings to the model The positional ...
figs_and_nuts's user avatar
2 votes
3 answers
61 views

How can we use a transfomer model with new data if we still don't have the output?

Transformer models are trained using inputs and outputs. They are both embedded and encoded and used to train multi-head attention mechanisms... But how can we use a transformer model to predict new ...
skan's user avatar
  • 185
0 votes
0 answers
9 views

Deep Learning Methods for Video Classification

I'm working on a dataset with ~300 videos that last from 9 to 13-minute interviews of each subject and it has all the personality-related metadata that was collected during initial surveys. Which Deep ...
TheBiometricsGuy's user avatar
1 vote
1 answer
25 views

A good set of datasets/models for testing an NLP technique

I am a machine learning researcher who up until this point has primarily worked on Computer Vision problems. However, I have an idea for an NLP technique involving a novel Transformer architecture, ...
Anonymous's user avatar
1 vote
1 answer
52 views

Why is T5 often used in text-to-data for text prompt encoders?

In the text-to-data(music, image, audio, etc.) generative AI field, one method of encoding text prompts is to use pre-trained language models. Such an approach was used in research on Moûsai [1] and ...
NakataKoo's user avatar
0 votes
1 answer
33 views

Neural regression predictions all around the mean of target

I have a transformer regression model and some data about last users transactions (categorical and numerical). My target has exponential distribution with mean aroud 10e4 and also zero-inflated, so I ...
CoolHumphy's user avatar
0 votes
0 answers
198 views

How to get Llama-2 Rotary Embeddings?

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the ...
Christian01's user avatar
0 votes
0 answers
15 views

Errors in results after saving model vs using directly from memory

I am trying to save a Fine Tuned model using trainer.save_model() but after I load the saved_model it just responds with the input back again and does not give any ...
Nikhil Verma's user avatar
0 votes
0 answers
42 views

Different generated patches from original image using vision transformer (ViT)

I am using ViT for image classification, I scaled images in range of [-1,1], and I also padded images. Then, I used the following code to see the original image and generated patches, but the output ...
Zara Nz's user avatar
0 votes
1 answer
41 views

Some fundamental questions about Transformer

In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e., \begin{align} y^t = x^...
Keyflux's user avatar
  • 101
2 votes
0 answers
44 views

CLIP Visual Transformer image encoder

I was doing some experiments with the CLIP's visual transformer encoder output (clip-ViT-B-32). So basically given the same scene or image, it should output almost ...
Tina J's user avatar
  • 217
0 votes
0 answers
41 views

Swin Transformer Relative Position Biases

I was reading the swin transformer paper and looking at the github implementation, i noticed that when calculating the relative position bias the input to the log function before the CPB MLP is scaled ...
jack's user avatar
  • 1
0 votes
1 answer
31 views

Tensorflow diagram for attention mechanism

I was reading the tutorial from tensorflow on the transformer model, however, when they explain the transformer model, they display such a picture : which I don't understand. What do the ingoing ...
edamondo's user avatar
  • 101
1 vote
0 answers
45 views

Deploying a model with GPU and pay-per-inference

I may have the wrong stack exchange. If that's the case, could someone point me to a stack that could help with this. Anyways... My backend employs a sentence transformer model from HuggingFace. Since ...
AlwaysLearning's user avatar
0 votes
0 answers
59 views

Sum of vector sentence embeddings vs. paragraph embedding

I have been experimenting with the all-MiniLM-L6-v2 model for computing 384-dimensional vector embeddings for text paragraphs. The following code compares the embedding computed for a paragraph with ...
AlwaysLearning's user avatar
0 votes
0 answers
17 views

How can BERT/Transformer models accept input batches of different sizes?

I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input. How is that possible? I thought we ...
PS1's user avatar
  • 101
1 vote
2 answers
451 views

how do we adapt LLM token embeddings with custom vocab

Hi im just getting started with understanding transformer based models and I am not able to find how the token embeddings are arrived at?. there are multiple tokenization approaches and multiple ...
dasman's user avatar
  • 111
0 votes
0 answers
24 views

Representation of 1D time series for use in transformers

I am working on a transformer where the encoder is given a sentence. However, the decoder is given 1D time series data (so, a single 1xm row vector). Given that the encoder is given n 350 dimensional ...
Avatrin's user avatar
  • 131
0 votes
0 answers
15 views

Differences between the two ViT-based models: GLP and DPT?

I've been learning to do some depth estimation tasks so I came across the GLP model (https://arxiv.org/pdf/2201.07436.pdf) which is suggested on huggingface website. I'm new to machine learning, so ...
user113988's user avatar
2 votes
2 answers
2k views

Understanding alpha parameter tuning in LORA paper

I was reading the LORA paper https://arxiv.org/pdf/2106.09685.pdf a thing I don’t understand is section 4.1, where the updates are updated by alpha, where alpha is a constant in r. It is said that ...
jpotwor's user avatar
  • 21
0 votes
1 answer
83 views

Do transformers output probabilities depend only on previous tokens?

In a transformer, the target sequence shifted right is placed in the input of the transformer decoder, and a upper triangular mask is provided so that the attention layer will not depend "tokens ...
blallo's user avatar
  • 3
0 votes
0 answers
601 views

Unsupervised fine tuning of Code LLMs

How to prepare code data to fine tune a code LLM in an unsupervised way or is it even possible? For example: Task: Code summarization with custom code base (with no summaries) Let's assume that this ...
Maximos's user avatar
2 votes
2 answers
264 views

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models. In the original "Attention is All You Need&...
mutli-arm-bandit's user avatar
0 votes
0 answers
271 views

Adapting a BERT-based model from HuggingFace for NER (named entity recognition) and RE (relation extraction)?

Context: NER (named entity recognition) and RE (relation extraction) from sentences obtained from radiology reports (medical text). There is a BERT-based model from HuggingFace I would like to use for ...
Pablo Messina's user avatar
0 votes
0 answers
25 views

Using ESM2 to predict GRAB sensor performance

I am an undergrad working in a neuroscience lab, and my PI has tasked me with using ESM2(https://github.com/facebookresearch/esm) to predict GPCR Activation Based (GRAB) sensor performance for in vivo ...
Garrett Goodrum's user avatar
0 votes
0 answers
21 views

What Can Prevent Time-Series Prediction Model From Learning Trend?

I am building an encoder-decoder prediction model based on this paper: https://www.sciencedirect.com/science/article/pii/S0952197623001483 It is made of a transformer encoder and a 1D CNN Decoder. The ...
LaTate's user avatar
  • 1
0 votes
0 answers
34 views

Are there other "interactive" non-linear neural network layers besides self-attention layer?

In the self-attention layer $$ \operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V $$ $Q$, $K$ and $V$ are all linear with respect to embedding vectors $x$, ...
DeLorean88's user avatar
2 votes
3 answers
221 views

What does it mean order of input sequence does not matter for transformer self-attention head?

The need for positional encoding in transformer models is justified by permutation invariance of self-attention heads, because, without it, transformer wouldn't have any mechanism to take into account ...
DeLorean88's user avatar
0 votes
1 answer
33 views

How quickly can a transformer self-heal if you wipe out one of its layers?

Say we have a fully-trained N-layer transformer model (encoder-only, decoder-only, or encoder-decoder), with embedding dimension ...
Darren Cook's user avatar
0 votes
0 answers
25 views

What dimensional reduction and similarity score work for sentence embeddings created using sentence transformers

I am clustering sentence embeddings for log files, and find anomalies. So, when I create sentence embeddings for logs using sentence transformers. It will create vector of fixed length, which somehow ...
Glinty's user avatar
  • 23

1
2 3 4 5
9