Skip to main content

Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
2 votes
0 answers
21 views

Use text embeddings to map job descriptions to ESCO occupations

I'm trying to build a model to map job descriptions to ESCO occupations which is a taxonomy for job titles. Every ESCO occupations have a title, a description and some essential skills. Ideally I ...
GanaelD's user avatar
  • 21
0 votes
0 answers
22 views

Why is positional encoding preferable over adding additional features for transformer models?

Why is information about the position not added as an additional feature? I read in forums that the only reason would be length-based overfitting, but I couldn't find a reliable source for that. What ...
kot's user avatar
  • 11
0 votes
0 answers
19 views

Insights about W0rd2Vec

As per my knowledge, Word2Vec is belongs to non-contextual embedding technique. this have only semantic relationship between words. We can implement Word2Vec, either in CBoW or skip-gram model. but i ...
Tovlk's user avatar
  • 43
0 votes
0 answers
21 views

Is the score function form of ALiBi, a positional encoding in Deep Learning, always lower triangular?

I have a question about the score function of ALiBi (Attention with Linear Biases), which is a positional encoding method introduced by the following paper: TRAIN SHORT, TEST LONG: ATTENTION WITH ...
shx's user avatar
  • 101
0 votes
0 answers
8 views

Do LSTM, GRU and Transformer models with less layers and units perform better than larger models when classifying short text sequences?

I am working with a Kaggle dataset with short Twitter messages as text input. I made a copy here. When testing LSTMS, GRUs, bi-directional versions of the GRUs, and the Encoder layers of a Transformer ...
Joachim Rives's user avatar
1 vote
1 answer
76 views

Why Transformer applies Dropout after Positional Encoding?

Why Transformers applies Dropout after Positional Encoding? Attention Is All You Need Not sure what is the benefit of removing 10% of tokens in a sequence by default. Read Why use dropout in ...
mon's user avatar
  • 711
0 votes
0 answers
16 views

What is the advantage of positional encoding over using additional features?

Popular models such as the transformer model use positional encoding on existing feature dimensions. Why is this preferred over adding more features to the feature dimension of the tensor which can ...
kot's user avatar
  • 11
0 votes
0 answers
9 views

Character-wise accuracy for image-to-text models

is it possible to enforce image-to-text models like ViT or a simple CNN+Transformer to achieve character-wise accuracy? Here's the context of my project: I am developing a model to extract some ...
CarlV's user avatar
  • 1
0 votes
1 answer
28 views

What's the purpose of using MLM when pretraining?

If BERT is a stack of transformer encoders, and the encoder already operates bidirectionally, understanding both left and right contexts and generating contextual embeddings, what is the purpose of ...
user avatar
0 votes
0 answers
11 views

can decoder only large language model be fine tuned to perform well at semantic similarity search?

BERT based models are Encoder only which are well suited for text classification, and Semantic Text similarity search (If fine-tuned via sBERT). I want to know whether decoder only models like Llama2, ...
haneulkim's user avatar
  • 469
0 votes
1 answer
31 views

How do transformer-based architectures generate contextual embeddings?

How do transformer-based architectures, such as Roberta, etc., generate contextual embeddings? The issue is, I haven't found any articles that explain this process.
user avatar
0 votes
0 answers
21 views

Approach for Multi-class Classification of texts

I'm trying to do a project where I have paragraphs and I need to classify them into multiple labels. The dataset is around 40k rows with labels. I understand there is no one right approach but should ...
Shaurya Uniyal's user avatar
0 votes
0 answers
13 views

what is the main difference between ROUGE and BLUE?

Both (ROUGE, BLUE) are useful to find the similarity between machine generated summary and reference summary. what is the main difference?
Tovlk's user avatar
  • 43
0 votes
0 answers
9 views

Reducing language bias for text classification, transformer model

I am working on a text classification model predicting classes for text. We have languages from many parts of the world and some of our classes are dominated by specific languages. The model we are ...
Carl Rynegardh's user avatar
0 votes
1 answer
17 views

Could You Suggest Me Some Details of Realizing This LLM?

I mean this hypothetical LLM: https://twitter.com/RokoMijic/status/1663299142431432704 I'm trying to figure out how the neural network (let's abstract from the data) can be realized. I understand that:...
avpol's user avatar
  • 21
0 votes
0 answers
56 views

Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:3! in DPOTrainer with ec2 G5 12X Large

...
Sandun Tharaka's user avatar
2 votes
1 answer
38 views

Effect of Sequential Data Quality Variation on Transformer Model Training: Seeking Insights and Experiences

I'm exploring the training efficiency of transformer models against the backdrop of data quality sequencing. Specifically, I ponder whether arranging unlabeled data by presumed quality affects ...
Lukas N.P. Egger's user avatar
1 vote
0 answers
38 views

Training Models Directly with Transformer Attention Weights: A Viable Strategy?

I'm currently using pre-trained transformers to extract embeddings for sequence analysis, which are then used in downstream tasks. My process involves using the extracted embeddings as features for ...
pparker's user avatar
  • 392
1 vote
1 answer
37 views

Does Google DeepMind's Gemma 7B models specs have inconsistent dimensions?

In Google DeepMind's Gemma technical paper (https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf), the 7B Gemma model specs are given as d_model = 3072, num_heads = 16 and head_size = ...
Unmesh's user avatar
  • 11
0 votes
1 answer
42 views

Where the term `cross-attention` is first used? (couldn't find the term in attention is all you need paper)

I am looking for the paper that first used the term cross-attention. I carefully read the paper ...
Jiho Choi's user avatar
  • 125
1 vote
1 answer
36 views

How can I use contextual embeddings with BERT for sentiment analysis/classification

I have a BERT model which I want to use for sentiment analysis/classification. E.g. I have some tweets that need to get a POSITIVE,NEGATIVE or NEUTRAL label. I can't understand how contextual ...
average_discrete_math_enjoyer's user avatar
1 vote
1 answer
32 views

Aside from trial and error, how do I select the number of layers and unit counts for LSTMS, GRUs, and Transformer units for text and time series?

When deciding on the number of units and layers for text processing or time-series prediction I rely heavily on trial and error. First, I look for a reference or paper on the topic such as the white ...
Joachim Rives's user avatar
0 votes
0 answers
7 views

Gradually increasing CPU load on using sentence embeddings model with kmeans

I am having a ML based production application, using flask, deployed on GCP server using gunicorn workers. In each incoming request, a text sentence is received. It is using sentence transformers (All-...
racdev's user avatar
  • 1
1 vote
0 answers
101 views

Why do the Llama 2 weights have eight different files?

I downloaded the weights for Llama 2 (70B-chat). This process created a folder titled "llama-2-70b-chat," which contained 8 files titled consolidated.00.pth, consolidated.01.pth, and so on ...
jskattt797's user avatar
1 vote
0 answers
53 views

How can I use Time-GPT for pretraining my model

I am mentioning Time-GPT here as a placeholder example. It can be any pretrained model. Suppose I have a dataset that requires some time series prediction. How can I leverage a well-trained model and ...
Mohammad Mosiur's user avatar
0 votes
0 answers
21 views

Using Transformers on a seq2seq task with sparse labels

I'm new to the ML world and would like to ask for architecture advice for a project I'm building. I want to detect a certain event throughout an audio. For example, if the audio is divided into 10 ...
TigerHix's user avatar
  • 101
0 votes
0 answers
7 views

Strategies for Encoding Large Datasets in Symbolic Music Generation for BERT-type Model

I am creating a BERT-type model for symbolic music generation. An observation of my database is a musical piece. Actually, is a "viewpoint" of the piece: ...
Kikolo's user avatar
  • 101
1 vote
1 answer
126 views

Is vision transformer (ViT) always better than CNN?

The paper - AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE proposed vision transformer and outperformed CNN-based models in many cases. When it comes to sequential data, we ...
Chuck Liu's user avatar
0 votes
1 answer
44 views

Why the standard deviation of the BERT weight initialization is 0.02 by default

The purpose of weight initialization in the neural network is to keep the variance of calculation output in the layers to 1.0, and it depends on the calculations involved in the layers. Initializing ...
mon's user avatar
  • 711
0 votes
1 answer
23 views

Getting rid of the warning "The following columns ... have been ignored" and "ValueError: batch_size should be a positive integer value ..."

I train a fine-tuning model with the PyTorch Trainer class: ...
questionto42's user avatar
0 votes
0 answers
9 views

How to calibrate IMU for large scale deployments possibly using deep neural network

We were testing our visual SLAM algorithm on robots. We were getting poor performance. Then we calculated wite noise and random walk parameters (using kalibr) for the IMU and used it in our algorithm ...
Mahesha999's user avatar
1 vote
1 answer
43 views

Annotated Transformer - Why x + DropOut(Sublayer(LayerNorm(x)))?

Please clarify if the Annotated Transformer Encoder LayerNorm implementation is correct. Transformer paper says the output of the sub layer is ...
mon's user avatar
  • 711
0 votes
1 answer
48 views

Getting a free and unknown answer to a question against a fine-tuned text generation model trained on many essays and their few questions and answers

Aim I want to fine-tune a text generation model with essays of changing size and then ask each of these input texts a few questions. I already have a wider range of question-answer pairs at hand for ...
questionto42's user avatar
0 votes
1 answer
79 views

Outdated Transformers TextDataset class drops last block when text overlaps. Replace by datasets Dataset class as input of Trainer train_dataset?

Why I try to replace the transformers TextDataset class with datasets Dataset class I stumbled upon this when I tried to make ...
questionto42's user avatar
0 votes
1 answer
251 views

Transformers Trainer: "RuntimeError: module must have its parameters ... on device cuda:6 (device_ids[0]) but found one of them on device: cuda:0"

I ask this since I could not fix it with the help of: Stack Overflow RuntimeError: module must have its parameters and buffers on device cuda:1 (device_ids[0]) but found one of them on device: cuda:2 ...
questionto42's user avatar
0 votes
0 answers
17 views

How to perform inference on a finetuned falcon 7b model fine tuned on open assistant dataset

I finetuned a falcon 7b model on the open assistant dataset using the official colab notebook provided by huggingface at https://colab.research.google.com/drive/1BiQiw31DT7-cDp1-0ySXvvhzqomTdI-o?usp=...
Saket Vempaty's user avatar
0 votes
0 answers
16 views

How are GPT2 token embedding vectors processed internally?

I am experimenting with the GPT2-XL model and trying to understand the internal structure. While I understand most of the components and how they affect the size of the activation tensors (such as ...
John Doe's user avatar
1 vote
1 answer
40 views

Falcon-7B llm giving random output

I am using a falcon 7B model for a chatbot without any finetuning with the following code ...
Saket Vempaty's user avatar
0 votes
0 answers
27 views

Application of transformers to tabular data

Is anyone using transformer based models for tabular data in real data science jobs as of 2024 I mean models like Tabnet Tabtransformer ARM-net SAINT FT-Transformer Non-Parametric Transformers I got ...
Ggjj11's user avatar
  • 216
2 votes
0 answers
48 views

Why does cross-attention in an NMT decoder use the encoder embeddings as values?

In the Vaswani 2017 paper introducing encoder-decoder transformers, the cross-attention step in the decoder is visualised as follows: Because keys and values are always taken to be equal, this figure ...
Mew's user avatar
  • 233
0 votes
0 answers
46 views

Why does scaled dot-product attention use softmax?

I am trying to understand the reasoning behind the Transformer architecture. In "Attention is all you need", the weights for the scaled dot-product attention is defined as the scaled dot-...
Reinis Mazeiks's user avatar
0 votes
0 answers
7 views

How to label a dataset of text pairs to use it as a universal one for calculating the precision@k metric for different models?

I am facing a semantic search problem. I am fine tuning different NLU models and i want to use precision@k as my main metric. Is it possible to label a dataset of text pairs to use it as a universal ...
Ir8_mind's user avatar
  • 183
0 votes
0 answers
47 views

Could someone help with fine-tuning dolphin-2.2.1?

Could someone help with fine-tuning dolphin-2.2.1? I have a problem with training: my train\loss - 0 and validation\loss - 0.000... after 800-1000 steps and this is overfitting ...
kabba62's user avatar
6 votes
2 answers
735 views

Cross-attention mask in Transformers

I can't fully understand how we should create the mask for the decoder's cross-attention mask in the original Transformer model from Attention Is All You Need. Here is my attempt at finding a solution:...
ИванКарамазов's user avatar
0 votes
1 answer
39 views

Using activations at a specific layer as an input for an LLM such as OPT-350m

I'm working with the OPT-350m model and aiming to utilize embeddings from different layers as inputs for generating data. I've encountered issues when trying to feed these embeddings back into the ...
Talia Seada's user avatar
0 votes
0 answers
10 views

The using of golden dataset in Augmented SBERT Training

I use the training strategy of Augmented SBERT (Domain-Transfer). In the code example they use the golden-dataset (STSb) for the training evaluator. Here two code snippets of the example of sentence-...
Christian01's user avatar
0 votes
0 answers
26 views

Interpretation of Evaluation Values of Augmented SBERT Training with EmbeddingSimilarityEvaluator()

I train a BI-Encoder to get an Augmented SBERT and I get a final training result. How can I interpret the following output of the final training result? ...
Christian01's user avatar
1 vote
0 answers
18 views

How does a spare mixture of experts route each token separately to an expert?

As far as I can understand the routing mechanism in the the sparsely-gated mixture-of-experts layer routes each token to its own expert. Considering that the output of the attention layer is a three ...
gamer bg's user avatar
0 votes
1 answer
63 views

Do we really need a very large dataset to train GPTs?

Do we really need a very large dataset to train GPTs? If this dataset is not big, won't GPT work well? Or will it still work better than conventional learning models in this situation? And is it ...
nisar's user avatar
  • 1
0 votes
1 answer
57 views

RuntimeError: mat1 and mat2 shapes cannot be multiplied (25x7 and 1x512)

I am dealing with multivariate time series forecasting using Transformers. below is my code step by step: After some preprocessing and windowing time series dataset … 1- Creating Mask function ...
Amir's user avatar
  • 1

1
2 3 4 5
10