Skip to main content
15 votes
Accepted

How to determine feature importance in a neural network?

Don't remove a feature to find out its importance, but instead randomize or shuffle it. Run the training 10 times, randomize a different feature column each time and then compare the performance. ...
scholle's user avatar
  • 174
12 votes

Why do we need to add START <s> + END </s> symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals ...
David Marx's user avatar
  • 3,258
11 votes
Accepted

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
noe's user avatar
  • 27k
6 votes

How to determine feature importance in a neural network?

Linking to the same paper as @scholle but explaining the process differently (book and paper). You do not need to train the model multiple times. The algorithm described in the links above require a ...
rodrigo-silveira's user avatar
6 votes
Accepted

How do attention mechanisms in RNNs learn weights for a variable length input

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and ...
Esmailian's user avatar
  • 9,352
5 votes

How/What to initialize the hidden states in RNN sequence-to-sequence models?

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural ...
Mati K's user avatar
  • 95
5 votes

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous ...
Jindřich's user avatar
  • 1,751
5 votes
Accepted

How does the Transformer predict n steps into the future?

The Transformer is a seq2seq model. At training time, you pass to the Transformer model both the source and target tokens, just like what you do with LSTMs or GRUs with teacher forcing, which is the ...
noe's user avatar
  • 27k
4 votes

Predict output sequence one at a time with feedback

I assume, that your data looks somehow like this (a pair of question and answer) ...
Marmite Bomber's user avatar
4 votes

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here. my original code is: ...
Simon Ren's user avatar
4 votes

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

Q, K, V vectors are trained with standard backpropagation. All trainable parameters are initialized at random, and then adjusted step by step with a Gradient Descent algorithm. Surprisingly, they are ...
Leevo's user avatar
  • 6,265
4 votes
Accepted

How to use BERT in seq2seq model?

In principle, it is possible to reuse the special tokens as you describe. However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain ...
noe's user avatar
  • 27k
3 votes

How to determine feature importance in a neural network?

You can do this sort of thing using SHAP, it looks at permutation importance as well.
Brian Droncheff's user avatar
3 votes

Methods for learning with noisy labels

First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples. In my opinion "noisy" is when positive and negative ...
Erwan's user avatar
  • 25.5k
3 votes

Transformer seq2seq model and loading embeddings from XLM-RoBERTa

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and ...
Jindřich's user avatar
  • 1,751
3 votes
Accepted

Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder

About the need for tgt_key_padding_mask While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before ...
noe's user avatar
  • 27k
3 votes
Accepted

Requirements for variable length output in transformer

The Transformer has the inherent ability to generate variable-length sequences, you don't need to do anything special. The output of the Transformer decoder is always the same length as the input of ...
noe's user avatar
  • 27k
3 votes
Accepted

How to select the optimal beam size for beam search?

Large beam sizes do not lead to improvements but to degradation in the generated text quality, as described in the article Empirical Analysis of Beam Search Performance Degradation in Neural Sequence ...
noe's user avatar
  • 27k
2 votes
Accepted

Training Encoder-Decoder using Decoder Outputs

I would like to know if it's possible to train a decoder by feeding its predictions at time, t-1, into the input at time-step t. Yes, it is possible to do it. But I don't see why you would do it. ...
pcko1's user avatar
  • 3,950
2 votes
Accepted

Do we really need <unk> tokens?

The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the ...
n1k31t4's user avatar
  • 14.9k
2 votes

Very long sequence in neural networks

You can indeed use the ability of recurrent network like LSTM to handle the varying length problem. But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a ...
Adrien D's user avatar
  • 1,113
2 votes

Can Sequence to sequence models be used to convert code from one programming language to another?

Yes, it is possible to convert code from one programming language to another using sequence-to-sequence neural networks. Sequence-to-sequence can learn to translate anything from anything if there are ...
Brian Spiering's user avatar
2 votes
Accepted

Why do position embeddings work?

The token embeddings are not fixed, they are learned. Therefore, during training, the value learned for the token embeddings is intrinsically one that is useful after adding it up with the positional ...
noe's user avatar
  • 27k
2 votes

One-hot encode multi-class multi-label sequences

You may need to try cat2vec which converts categorical features into vector representation using Word2Vec approach. Check also this link for multi-feature inputs into LSTM. For the target y, one-hot ...
Mulugeta Weldezgina's user avatar
2 votes
Accepted

How can I feed BERT to neural machine translation?

I find this way of using BERT in my translation system and it allows me to load and use more data to train my model. I got a memory error when I want to use more data like 100k for my task. and I came ...
Hamed's user avatar
  • 39
2 votes

Sentences language translation with neural network, with a simple layer structure (if possible sequential)

Machine translation using traditional neural architecture (seq2seq models) had various issues due to rare-words, low accuracy and slow translation [1]. Even after using various mechanisms like ...
Vikas Bhandary's user avatar
2 votes

Methods for learning with noisy labels

There is a python package created exactly for this purpose of finding label errors and training ML models robustly and reliably even when your data has issues or you have noisy labels: https://github....
cgnorthcutt's user avatar
2 votes

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

As Jindřich has said, Q, K, V come from previous computations, they are not trained directly with backpropagation. However, the weights $W_i^Q, W_i^K, W_i^V$ are trained directly with backpropagation. ...
Kyle Nickerson's user avatar
2 votes

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

I'm a PhD student in natural language processing, and I hope I can clear up some of the terminology used in previous answers to this question in a way that's helpful for full understanding. To clarify ...
Pro Q's user avatar
  • 195
2 votes

Why does an attention layer in a transformer learn context?

To provide a simplistic and less mathematical reasons. You can assume like this: In a simple feed-forward neural network (a black-box of course), you shall learn the set of weights, learning a ...
Ashwin Geet D'Sa's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible