Questions tagged [attention-mechanism]
The attention-mechanism tag has no usage guidance.
16
questions
0
votes
1answer
26 views
Attention Mechanism: Why use context vector instead of attention weights?
In attention, the context vector ($c$) is derived from the sum of the attention weights ($\alpha$) multiplied by the encoder hidden states ($h$), where the weights are obtained by multiplying the ...
0
votes
0answers
25 views
ValueError: Dimensions must be equal, but are 256 and 12 for 'attention_layer/MatMul_1' (op: 'MatMul') with input shapes: [?,256], [12,256]
I'm working on a sequence-to-sequence approach using LSTM and a VAE with an attention mechanism.
...
0
votes
0answers
87 views
ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)
I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism.
...
4
votes
2answers
1k views
What is positional encoding in Transformer model?
I'm new to ML and this is my first question here, so sorry if my question is silly.
I'm trying to read and understand the paper Attention is all you need and in it, there is a picture:
I don't know ...
1
vote
0answers
57 views
What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?
As stated in the question above..is there a difference between attention and self attention mechanism ? Also additionally can anybody share with me tips and tricks about how self attention mechanism ...
7
votes
1answer
1k views
Can BERT do the next-word-predict task?
As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
0
votes
1answer
131 views
What is the reason for the speedup of transformer-xl?
The inference speed of transformer-xl is faster than transformer.
Why?
If state reuse is the reason, so it is compared by two 32seq_len + state-reuse vs one 64seq_len + no-state-reuse?
2
votes
0answers
115 views
Keras value error: Operands could not be broadcast with with shapes(100,100) - GRU
I am trying to use Hierarchical Attention Networks for classification of news articles using 20 newsgroup dataset that i downloaded from the internet. I came across this code of the implementation and ...
1
vote
1answer
239 views
Variable input/output length for Transformer
I was reading the paper "Attention is all you need" (https://arxiv.org/pdf/1706.03762.pdf ) and came across this site
http://jalammar.github.io/illustrated-transformer/ which provided a great ...
1
vote
0answers
470 views
How to train tensorflow's transformer model on my own data?
https://github.com/tensorflow/models/blob/master/official/transformer has an implementation of transformer model.
I want to train the model on my own data(consisting of two files, src.txt, and tgt.txt)...
2
votes
2answers
503 views
Keras Attention Guided CNN problem
I am working on a CNN for XRay image classification and I can't seem to be able to properly train it. I am trying to implement the following paper in Keras: https://arxiv.org/pdf/1801.09927.pdf
In ...
0
votes
1answer
70 views
Why and how BERT can learn different attentions for each head?
https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77
I read the blog above. It visualizes that different color/head has different ...
0
votes
0answers
26 views
Sub-Object Attention models
Questions first:
I need help to focus myself on the most relevant attention model papers (Attention to Attention if you will). Where should I start?
Have you heard of attention models that focus on ...
1
vote
0answers
97 views
How do I implement an attention mechanism for convolutional neural network in Keras?
I have a convolutional neural network in Keras on which I'd like to add an attention mechanism? Has anyone done this? It seems Keras doesn't have an in-built attention mechanism and the ones I've ...
1
vote
0answers
726 views
Why does Position Embeddings work?
In the papers "Convolutional Sequence to Sequence Learning" and
"Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order ...
6
votes
1answer
512 views
How do attention mechanisms in RNNs learn weights for a variable length input
Attention mechanisms in RNNs are reasonably common to sequence to sequence models.
I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output ...
