# Tag Info

• 131

### What is the positional encoding in the transformer model?

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take ...
Accepted

### Transformer model: Why are word embeddings scaled before adding positional encodings?

This is specified in the original Transformer paper, at the end of section 3.4: Transcription: 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned embeddings ...
• 16.3k
Accepted

### Variable input/output length for Transformer

Your understanding is not correct: in the encoder-decoder attention, the Keys and Values come from the encoder (i.e. source sequence length) while the Query comes from the decoder itself (i.e. target ...
• 16.3k

### Why does the transformer positional encoding use both sine and cosine?

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input. In order not to use a sequence of integers (1, ...
• 1,772

### How does attention mechanism learn?

From the Amazing Blog - FloydHub Blog- Attention Mechanisms Attention Mechanisms Attention takes two sentences, turns them into a matrix where the words of one sentence form the columns, and the ...
• 2,466
Accepted

### How do attention mechanisms in RNNs learn weights for a variable length input

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and ...
• 8,667

### What is the positional encoding in the transformer model?

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the ...
• 61
Accepted

### Does BERT use GLoVE?

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called ...
• 1,426
Accepted

### In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have ...
• 16.3k
Accepted

### Attention for time-series in neural networks

It is an interesting question. I would not completely agree with you though when you say that most time-series models dont use attention. However there is not as much documentation available on the ...
• 683

### How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous ...
• 1,426
Accepted

### Can BERT be used for predicting words?

pip install transformers Then try this ...

### Transformer model: Why are word embeddings scaled before adding positional encodings?

Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized ...
• 615

### What is the advantage of positional encoding over one hot encoding in a transformer model?

You are mixing two different concepts in the same question: One hot encoding: approach to encode $n$ discrete tokens by having an $n$-dimensional vectors with all 0's except one 1. This can be used ...
• 16.3k

### What is the advantage of positional encoding over one hot encoding in a transformer model?

The theoretical advantage should be that the network should be able to grasp the pattern from the encoding and thus generalize better for longer sentences. With one-hot position encoding, you would ...
• 1,426

### ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here. my original code is: ...

### Gumbel-Softmax trick vs Softmax with temperature

From a practical and theoretical perspective, when is it beneficial to incorporate Gumbel noise into a neural network, as opposed to just using Softmax with temperature? You don't necessarily ...
• 131
Accepted

### Transformer decoder output - how is it linear?

I'm not quite sure how's the decoder output is flattened into a single vector That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the ...
• 5,186
Accepted

### What reccent alternatives to LSTM are there for regression problems?

Transformers (aka "attention models") are being used in place of LSTMs in many areas, as they generally give better results, and/or are quicker to train. They can be used for regression ...
• 615

### What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

In attention mechanisms, you take an expectation of a representation of data V with respect to some probability mass function, thus computing the context vector, which is essentially a summary ...
• 121

### Gumbel-Softmax trick vs Softmax with temperature

For the softmax function, no matter what is the temperature, it is not the exact one-hot vector. If you could accept a soft version, it is good. However, if you choose the argmax to be the one, it is ...
• 21