Data Science

Questions tagged [attention-mechanism]

Ask Question

The tag has no usage guidance.

26 questions with no upvoted or accepted answers
Newest
Active
Bountied
Unanswered
Filter by
Sorted by
Tagged with
3
votes
1answer
2k views

SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors

I am writing Encoder-Decoder architecture with Bahdanau Attention using tf.keras with TensorFlow 2.0. Below is my code This is working with TensorFlow 1.15 but getting the error in 2.0. you can check ...
3
votes
0answers
700 views

How to train tensorflow's transformer model on my own data?

https://github.com/tensorflow/models/blob/master/official/transformer has an implementation of transformer model. I want to train the model on my own data(consisting of two files, src.txt, and tgt.txt)...
3
votes
0answers
2k views

Why does Position Embeddings work?

In the papers "Convolutional Sequence to Sequence Learning" and "Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order ...
2
votes
0answers
316 views

Keras value error: Operands could not be broadcast with with shapes(100,100) - GRU

I am trying to use Hierarchical Attention Networks for classification of news articles using 20 newsgroup dataset that i downloaded from the internet. I came across this code of the implementation and ...
1
vote
1answer
62 views

How do attention mechanism in CNN for images?

I read some attention mechanism papers, but I could not understand how it can be applied to an image (classification, detection, etc) using a CNN model. How does it affect the alignment scores and the ...
1
vote
0answers
22 views

how is the linear relation between positional encoding helping attention?

I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that ...
1
vote
1answer
26 views

Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task?

So I just used BERT pre-trained with Focal Loss to classify Physics, Chemistry, Biology and Mathematics and got a good f-1 macro of 0.91. It is good given it only had to look for the tokens like ...
1
vote
1answer
37 views

Is a dense layer required for implementing Bahdanau attention?

I saw that everyone adds Dense( ) layer in their custom Bahdanau attention layer, which I think isn't needed. This is an image from a tutorial here. Here, we are just multiplying 2 vectors and then ...
1
vote
1answer
544 views

How to add a Decoder & Attention Layer to Bidirectional Encoder with tensorflow 2.0

I am a beginner in machine learning and I'm trying to create a spelling correction model that spell checks for a small amount of vocab (approximately 1000 phrases). Currently, I am refering to the ...
1
vote
1answer
50 views

How to add attention mechanism to my sequence-to-sequence architecture in Keras?

Based on this blog entry, I have written a sequence to sequence deep learning model in Keras: ...
1
vote
1answer
29 views

Attention network without hidden state?

I was wondering how useful the encoder's hidden state is for an attention network. When I looked into the structure of an attention model, this is what I found a model generally looks like: ...
1
vote
2answers
88 views

What are good toy problems for testing Transformer architectures?

I am testing various variants for Transformers and Transformer architectures. But training on full language tasks is a rather time consuming affair. What are good toy problems to test if a transformer ...
1
vote
1answer
31 views

Attention to multiple areas of same sentence

Lets consider some sentences below: "Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too" "Can company1 buy company2? ...
1
vote
0answers
139 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
1
vote
0answers
12 views

Training a model for Single Image Super Reoslution

I'm trying to implement the Attention-based approach for SISR paper. However, during something odd happens. The MAE for the first output of the model is very small. But as the training progresses, the ...
1
vote
0answers
68 views

How to Visualize Graph Attention

I am quite new to the concept of attention. I am working with graph data and running graph convolution on it to learn node level embedding first. Then an attention layer to aggregate the nodes to ...
0
votes
0answers
16 views

Basic of the attention mechanism

Background Having gone through articles. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Attention in Neural Networks Visualizing A Neural Machine Translation Model (...
0
votes
0answers
10 views

Using Transcoder Model for language to language conversion

I have a problem statement like Converting deprecated code into a modern version of the same language. I'm currently converting with a custom Rule-based engine. But the modern version of the language ...
0
votes
2answers
39 views

Understanding Transformer's Self attention calculations

I was going through this link: https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&utm_medium=demystifying-bert-groundbreaking-nlp-...
0
votes
0answers
19 views

Question about Relative-Position-Representation code

In https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/layers/common_attention.py In _relative_attention_inner method, which I think is one of the ...
0
votes
1answer
82 views

NLP Transformers - understanding the multi-headed attention visualization (Attention is all you need)

I am new to NLP and I just finished reading the paper "Attention is all you need". I'm struggling to understand the interpretability of the multi-headed attention, and specifically how these ...
0
votes
1answer
19 views

Predicting point sequence in image

My training set is a set of images (either 3 channel or 1 ofc i use only one type of channel). And the labels are a sequence of points in a specific order that i want to predict from the images. I am ...
0
votes
1answer
32 views

Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
0
votes
0answers
9 views

how many spectogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguisahble from humans) using the github https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
0
votes
1answer
95 views

Attention model with seq2seq over sequence

On the official tensorflow page there is one exmple of a decoder (https://www.tensorflow.org/tutorials/text/nmt_with_attention#next_steps): ...
0
votes
1answer
32 views

two different attention methods for seq2seq

I see two different ways of applying attention in seq2seq: (a) the context vector (the weighted sum of encoder hidden states) fed into the output softmax, as shown in the diagram below. The diagram ...