Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Attention model with seq2seq over sequence

On the official tensorflow page there is one exmple of a decoder (https://www.tensorflow.org/tutorials/text/nmt_with_attention#next_steps): ...
1
vote
1answer
22 views

Attention to multiple areas of same sentence

Lets consider some sentences below: "Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too" "Can company1 buy company2? ...
0
votes
1answer
40 views

How do Bahdanau - Luong Attentions use Query, Value, Key vectors?

In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and ...
2
votes
1answer
37 views

Why does the transformer positional encoding use both sine and cosine?

In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed. I am wondering why it needs to use both sine and cosine though instead of just ...
0
votes
2answers
36 views

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
1
vote
1answer
24 views

What is the feedforward network in a transformer trained on?

After reading the 'Attention is all you need' article, I understand the general architecture of a transformer. However, it is unclear to me how the feed forward neural network learns. What I learned ...
2
votes
1answer
50 views

Attention mechanism in Tensorflow 2

In the past days, I read up on the theory behind attention, when to apply it and what types there are. I think I have a decent first understanding of the concept, but now I would like to apply some of ...
10
votes
2answers
426 views

How does attention mechanism learn?

I know how to build an attention in neural networks. But I don’t understand how attention layers learn the weights that pay attention to some specific embedding. I have this question because I’m ...
0
votes
2answers
358 views

Why is the decoder not a part of BERT architecture?

I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT ...
1
vote
1answer
92 views

What is difference between attention mechanism and cognitive function?

How are attention mechanism used in different deep learning algorithms different from the cognitive function attention?
1
vote
0answers
515 views

SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors

I am writing Encoder-Decoder architecture with Bahdanau Attention using tf.keras with TensorFlow 2.0. Below is my code This is working with TensorFlow 1.15 but getting the error in 2.0. you can check ...
1
vote
0answers
42 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
1
vote
1answer
26 views

In Deep Learning, how many kinds of Attention exist? And what is the history of Attention models? [closed]

How many definitions of attention are commonly employed for Deep Learning tasks? That's what I've encountered up to now: Self-attention Bahdanau Luong Multi-Head (used in Transformers) Could you ...
1
vote
1answer
225 views

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
1
vote
0answers
11 views

Training a model for Single Image Super Reoslution

I'm trying to implement the Attention-based approach for SISR paper. However, during something odd happens. The MAE for the first output of the model is very small. But as the training progresses, the ...
0
votes
0answers
9 views

How can we use Attention Layer in a Encoder Decoder Model Architecture for Next Phrase Prediction?

I am currently building a Encoder Decoder Model for Next Phrase Prediction, but only using this model is just giving satisfactory results. So I chose to attach a Attention Layer to the same ...
0
votes
2answers
445 views

What is the advantage of positional encoding over one hot encoding in a transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, they used positional encoding with sin for even indices and cos for odd indices. In the paper (Section 3.5), they ...
0
votes
0answers
22 views

two different attention methods for seq2seq

I see two different ways of applying attention in seq2seq: (a) the context vector (the weighted sum of encoder hidden states) fed into the output softmax, as shown in the diagram below. The diagram ...
1
vote
0answers
26 views

How to Visualize Graph Attention

I am quite new to the concept of attention. I am working with graph data and running graph convolution on it to learn node level embedding first. Then an attention layer to aggregate the nodes to ...
0
votes
0answers
27 views

Hard Attention without RL or similar

I want to implement hard attention with Tensorflow in my neural network. To summarize my understanding of soft and hard attention, let's say we attend over $x_i$'s: In soft attention, we calculate ...
0
votes
0answers
13 views
0
votes
0answers
28 views

Pytorch Implementing Simple Attention using Dummy data

Hi I am trying to implement simple/General attention in Pytorch , So far the model seems to working , but what i am intersted in doing is getting the attention weights , so that i can visualize it . ...
5
votes
1answer
2k views

Gumbel-Softmax trick vs Softmax with temperature

From what I understand, the Gumbel-Softmax trick is a technique that enables us to sample discrete random variables, in a way that is differentiable (and therefore suited for end-to-end deep learning)....
0
votes
0answers
98 views

DA-RNN Keras implementation

Is there DA-RNN implementation with Keras or TensorFlow? If its a commented notebook it would be amazing https://arxiv.org/abs/1704.02971 here is the paper I am referring, I only found Torch ...
3
votes
0answers
476 views

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that : the positional encoding is summed to the word embeddings rather than concatenated to it. ...
0
votes
1answer
97 views

Attention Mechanism: Why use context vector instead of attention weights?

In attention, the context vector ($c$) is derived from the sum of the attention weights ($\alpha$) multiplied by the encoder hidden states ($h$), where the weights are obtained by multiplying the ...
0
votes
0answers
292 views

ValueError: Dimensions must be equal, but are 256 and 12 for 'attention_layer/MatMul_1' (op: 'MatMul') with input shapes: [?,256], [12,256]

I'm working on a sequence-to-sequence approach using LSTM and a VAE with an attention mechanism. ...
2
votes
0answers
656 views

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism. ...
22
votes
3answers
12k views

What is the positional encoding in the transformer model?

I'm new to ML and this is my first question here, so sorry if my question is silly. I'm trying to read and understand the paper Attention is all you need and in it, there is a picture: I don't know ...
1
vote
1answer
598 views

What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

As stated in the question above..is there a difference between attention and self attention mechanism ? Also additionally can anybody share with me tips and tricks about how self attention mechanism ...
11
votes
1answer
3k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
0
votes
1answer
253 views

What is the reason for the speedup of transformer-xl?

The inference speed of transformer-xl is faster than transformer. Why? If state reuse is the reason, so it is compared by two 32seq_len + state-reuse vs one 64seq_len + no-state-reuse?
2
votes
0answers
227 views

Keras value error: Operands could not be broadcast with with shapes(100,100) - GRU

I am trying to use Hierarchical Attention Networks for classification of news articles using 20 newsgroup dataset that i downloaded from the internet. I came across this code of the implementation and ...
5
votes
1answer
1k views

Variable input/output length for Transformer

I was reading the paper "Attention is all you need" (https://arxiv.org/pdf/1706.03762.pdf ) and came across this site http://jalammar.github.io/illustrated-transformer/ which provided a great ...
2
votes
0answers
659 views

How to train tensorflow's transformer model on my own data?

https://github.com/tensorflow/models/blob/master/official/transformer has an implementation of transformer model. I want to train the model on my own data(consisting of two files, src.txt, and tgt.txt)...
3
votes
2answers
1k views

Keras Attention Guided CNN problem

I am working on a CNN for XRay image classification and I can't seem to be able to properly train it. I am trying to implement the following paper in Keras: https://arxiv.org/pdf/1801.09927.pdf In ...
0
votes
1answer
84 views

Why and how BERT can learn different attentions for each head?

https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77 I read the blog above. It visualizes that different color/head has different ...
1
vote
0answers
134 views

How do I implement an attention mechanism for convolutional neural network in Keras?

I have a convolutional neural network in Keras on which I'd like to add an attention mechanism? Has anyone done this? It seems Keras doesn't have an in-built attention mechanism and the ones I've ...
3
votes
0answers
1k views

Why does Position Embeddings work?

In the papers "Convolutional Sequence to Sequence Learning" and "Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order ...
8
votes
1answer
783 views

How do attention mechanisms in RNNs learn weights for a variable length input

Attention mechanisms in RNNs are reasonably common to sequence to sequence models. I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output ...