Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
3 votes
1 answer
1k views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
0 votes
1 answer
142 views

How K and V are extracted from encoder output in transformer?

I was trying to understand transformer architecture from "Attention is all you need" paper. The paper shows following transformer architecture: How $K$ and $V$ is extracted from $512$ ...
1 vote
2 answers
215 views

Using BERT to extract a list of words and phrases from documents

I have a list of words and phrases (~3k items). What are my options to extract them from documents (~3M of job descriptions) with NLP? I do not have labeled data. For example my list of words and ...
0 votes
1 answer
81 views

Mix of time-dependent and constant features for a transformer

I'm using the transformer architecture to predict future time-points from previous time-points. Each item of the input sequence is a vector of ...
0 votes
1 answer
76 views

Predict the values of variable features over timestamps

HI i am having a dataset which contain timestamps and number of users at that timestamp. Each user has resource values which change per timestamp. How can i make predictions of number of users ...
0 votes
0 answers
7 views

Gradually increasing CPU load on using sentence embeddings model with kmeans

I am having a ML based production application, using flask, deployed on GCP server using gunicorn workers. In each incoming request, a text sentence is received. It is using sentence transformers (All-...
1 vote
0 answers
29 views

Why do the Llama 2 weights have eight different files?

I downloaded the weights for Llama 2 (70B-chat). This process created a folder titled "llama-2-70b-chat," which contained 8 files titled consolidated.00.pth, consolidated.01.pth, and so on ...
0 votes
1 answer
88 views

Compound and Complex Sentence Tokenization

I am trying to tokenize sentences of a document for aspect-based sentiment analysis. There are some sentences that consist of more than one topic. For example, " The touch screen is good but the ...
1 vote
1 answer
174 views

Does high number of output labels affect the performance of BERT and how to handle the class imbalance issue while doing multi text classification?

I am using BERT to do multiclass text classification. The number of output classes I have to predict from is: 116 and there is high degree of class imbalance that I see. We have the following kind of ...
0 votes
1 answer
82 views

Some fundamental questions about Transformer

In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e., \begin{align} y^t = x^...
0 votes
2 answers
296 views

Self-Attention Summation and Loss of Information

In self-attention, the attention for a word is calculated as: $$ A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>} $$ My question is why we sum over the ...
1 vote
1 answer
43 views

Is vision transformer (ViT) always better than CNN?

The paper - AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE proposed vision transformer and outperformed CNN-based models in many cases. When it comes to sequential data, we ...
38 votes
8 answers
10k views

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that : the positional encoding is summed to the word embeddings rather than concatenated to it. ...
1 vote
1 answer
361 views

How do I implement Dual-encoder model in Pytorch?

I am trying to implement the paper titled Learning Cross-lingual Sentence Representations via a Multi-task Dual-Encoder Model. Here the encoder and decoder share the same weights but I am unable to ...
1 vote
0 answers
42 views

How can I use Time-GPT for pretraining my model

I am mentioning Time-GPT here as a placeholder example. It can be any pretrained model. Suppose I have a dataset that requires some time series prediction. How can I leverage a well-trained model and ...
8 votes
4 answers
8k views

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
0 votes
0 answers
19 views

Using Transformers on a seq2seq task with sparse labels

I'm new to the ML world and would like to ask for architecture advice for a project I'm building. I want to detect a certain event throughout an audio. For example, if the audio is divided into 10 ...
0 votes
0 answers
6 views

Strategies for Encoding Large Datasets in Symbolic Music Generation for BERT-type Model

I am creating a BERT-type model for symbolic music generation. An observation of my database is a musical piece. Actually, is a "viewpoint" of the piece: ...
0 votes
1 answer
101 views

Transformers Trainer: "RuntimeError: module must have its parameters ... on device cuda:6 (device_ids[0]) but found one of them on device: cuda:0"

I ask this since I could not fix it with the help of: Stack Overflow RuntimeError: module must have its parameters and buffers on device cuda:1 (device_ids[0]) but found one of them on device: cuda:2 ...
0 votes
1 answer
38 views

Getting a free and unknown answer to a question against a fine-tuned text generation model trained on many essays and their few questions and answers

Aim I want to fine-tune a text generation model with essays of changing size and then ask each of these input texts a few questions. I already have a wider range of question-answer pairs at hand for ...
2 votes
1 answer
805 views

Fine-tuning a pre-trained LLM for question-answering

Objective My goal is to fine-tune a pre-trained LLM on a dataset about Manchester United's (MU's) 2021/22 season (they had a poor season). I want to be able to prompt the fine-tuned model with ...
1 vote
1 answer
253 views

Should weight distribution change more when fine-tuning transformers-based classifier?

I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation: ...
1 vote
2 answers
947 views

Fine-tuned MLM based RoBERTa not improving performance

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM. We took some sample data points (2M+) & fine-tuned ...
0 votes
1 answer
74 views

Library for Abstractive Summarization

Is there a Python library that supports abstractive summarization? (Excluding cloud-based models like GPT or ChatGPT). We can perform extractive summarization easily using the code below: ...
4 votes
2 answers
1k views

How is attention different from linear MLPs?

Each output for both the attention layer (as in transformers) and MLPs or feedforward layer(linear-activation) are weighted sums of previous layer. So how they are different?
0 votes
1 answer
14 views

Getting rid of the warning "The following columns ... have been ignored" and "ValueError: batch_size should be a positive integer value ..."

I train a fine-tuning model with the PyTorch Trainer class: ...
0 votes
1 answer
30 views

Why the standard deviation of the BERT weight initialization is 0.02 by default

The purpose of weight initialization in the neural network is to keep the variance of calculation output in the layers to 1.0, and it depends on the calculations involved in the layers. Initializing ...
1 vote
1 answer
1k views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
0 votes
1 answer
84 views

Incorporating structural information in a Transformer?

For a Neural Machine Translation (NMT) task, my input data has relational information. This relation could be modelled using a graphical structure. So one approach could be to use Graph Neural Network ...
0 votes
0 answers
7 views

How to calibrate IMU for large scale deployments possibly using deep neural network

We were testing our visual SLAM algorithm on robots. We were getting poor performance. Then we calculated wite noise and random walk parameters (using kalibr) for the IMU and used it in our algorithm ...
0 votes
2 answers
56 views

Bert model for document sentiment classification

I am trying to fine-tune a Bert model for sentiment analysis. Instead of one sentence, my inputs are documents (including several sentences) and I am not removing dots. I was wondering if is it okay ...
0 votes
1 answer
44 views

Outdated Transformers TextDataset class drops last block when text overlaps. Replace by datasets Dataset class as input of Trainer train_dataset?

Why I try to replace the transformers TextDataset class with datasets Dataset class I stumbled upon this when I tried to make ...
1 vote
1 answer
660 views

Why does a decoder generate all hidden states during inference?

Seems that in Vanilla transformers at least (a la AIAYN), during inference time, the hidden states are generated for all tokens in the input sequence, but only the last one is used to predict the next ...
1 vote
1 answer
35 views

Annotated Transformer - Why x + DropOut(Sublayer(LayerNorm(x)))?

Please clarify if the Annotated Transformer Encoder LayerNorm implementation is correct. Transformer paper says the output of the sub layer is ...
1 vote
1 answer
782 views

Is it possible to fine-tuning BERT by training it on multiple datasets? (Each dataset having it's own purpose)

BERT can be fine-tuned on a dataset for a specific task. Is it possible to fine-tune it on all these datasets for different tasks and then be utilized for these tasks instead of fine-tuning a BERT ...
1 vote
1 answer
307 views

Spatial positional encodings Vs Learned positional encodings(Object queries)

I have been trying to understand facebook's Detection transformer(DeTr) paper. Architecture Most of the explanation about the architecture is straightforward. I don't especially understand the ...
3 votes
1 answer
256 views

VQ-GAN understanding

I tried to understand how VQ-GAN works, but unfortunately I have not understood it. I tried to read some articles about it and watch a video. I believe a good and simple article will help me. You ...
0 votes
0 answers
17 views

How to perform inference on a finetuned falcon 7b model fine tuned on open assistant dataset

I finetuned a falcon 7b model on the open assistant dataset using the official colab notebook provided by huggingface at https://colab.research.google.com/drive/1BiQiw31DT7-cDp1-0ySXvvhzqomTdI-o?usp=...
0 votes
0 answers
14 views

How are GPT2 token embedding vectors processed internally?

I am experimenting with the GPT2-XL model and trying to understand the internal structure. While I understand most of the components and how they affect the size of the activation tensors (such as ...
1 vote
1 answer
38 views

Falcon-7B llm giving random output

I am using a falcon 7B model for a chatbot without any finetuning with the following code ...
0 votes
0 answers
17 views

Application of transformers to tabular data

Is anyone using transformer based models for tabular data in real data science jobs as of 2024 I mean models like Tabnet Tabtransformer ARM-net SAINT FT-Transformer Non-Parametric Transformers I got ...
0 votes
1 answer
634 views

ValueError: Exception encountered when calling layer "transformer" (type Transformer)

So I code a Transformers neural network that works as an ASR, it works, it trains good and saved the model as... model.save("savedmodel.model") The ...
1 vote
1 answer
243 views

How to further fine-tune a transformer NLP model on domain specific dataset, after general fine-tuning

I would like to fine-tune a pre-trained BERT-like model for a semantic similarity analysis task in the fashion of the SNLI/MNLI task (i.e. classify sentence pairs to "entailment" or "...
0 votes
2 answers
678 views

Understanding Transformer's Self attention calculations

I was going through this link: https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&utm_medium=demystifying-bert-groundbreaking-nlp-...
0 votes
1 answer
31 views

Using activations at a specific layer as an input for an LLM such as OPT-350m

I'm working with the OPT-350m model and aiming to utilize embeddings from different layers as inputs for generating data. I've encountered issues when trying to feed these embeddings back into the ...
2 votes
0 answers
42 views

Why does cross-attention in an NMT decoder use the encoder embeddings as values?

In the Vaswani 2017 paper introducing encoder-decoder transformers, the cross-attention step in the decoder is visualised as follows: Because keys and values are always taken to be equal, this figure ...
0 votes
0 answers
26 views

Why does scaled dot-product attention use softmax?

I am trying to understand the reasoning behind the Transformer architecture. In "Attention is all you need", the weights for the scaled dot-product attention is defined as the scaled dot-...
4 votes
2 answers
315 views

Cross-attention mask in Transformers

I can't fully understand how we should create the mask for the decoder's cross-attention mask in the original Transformer model from Attention Is All You Need. Here is my attempt at finding a solution:...
4 votes
2 answers
1k views

How to use is_split_into_words with Huggingface NER pipeline

I am using Huggingface transformers for NER, following this excellent guide: https://huggingface.co/blog/how-to-train. My incoming text has already been split into words. When tokenizing during ...
0 votes
0 answers
7 views

How to label a dataset of text pairs to use it as a universal one for calculating the precision@k metric for different models?

I am facing a semantic search problem. I am fine tuning different NLU models and i want to use precision@k as my main metric. Is it possible to label a dataset of text pairs to use it as a universal ...

1
2 3 4 5
10