Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [attention-mechanism]

The tag has no usage guidance.

6
votes
1answer
149 views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
0
votes
1answer
36 views

What is the reason for the speedup of transformer-xl?

The inference speed of transformer-xl is faster than transformer. Why? If state reuse is the reason, so it is compared by two 32seq_len + state-reuse vs one 64seq_len + no-state-reuse?
2
votes
0answers
41 views

Keras value error: Operands could not be broadcast with with shapes(100,100) - GRU

I am trying to use Hierarchical Attention Networks for classification of news articles using 20 newsgroup dataset that i downloaded from the internet. I came across this code of the implementation and ...
1
vote
0answers
52 views

Variable input/output length for Transformer

I was reading the paper "Attention is all you need" (https://arxiv.org/pdf/1706.03762.pdf ) and came across this site http://jalammar.github.io/illustrated-transformer/ which provided a great ...
1
vote
0answers
121 views

How to train tensorflow's transformer model on my own data?

https://github.com/tensorflow/models/blob/master/official/transformer has an implementation of transformer model. I want to train the model on my own data(consisting of two files, src.txt, and tgt.txt)...
2
votes
2answers
163 views

Keras Attention Guided CNN problem

I am working on a CNN for XRay image classification and I can't seem to be able to properly train it. I am trying to implement the following paper in Keras: https://arxiv.org/pdf/1801.09927.pdf In ...
0
votes
1answer
56 views

Why and how BERT can learn different attentions for each head?

https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77 I read the blog above. It visualizes that different color/head has different ...
0
votes
0answers
20 views

Sub-Object Attention models

Questions first: I need help to focus myself on the most relevant attention model papers (Attention to Attention if you will). Where should I start? Have you heard of attention models that focus on ...
1
vote
0answers
63 views

How do I implement an attention mechanism for convolutional neural network in Keras?

I have a convolutional neural network in Keras on which I'd like to add an attention mechanism? Has anyone done this? It seems Keras doesn't have an in-built attention mechanism and the ones I've ...
1
vote
0answers
277 views

Why does Position Embeddings work?

In the papers "Convolutional Sequence to Sequence Learning" and "Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order ...