14
votes
Accepted
How to determine feature importance in a neural network?
Don't remove a feature to find out its importance, but instead randomize or shuffle it.
Run the training 10 times, randomize a different feature column each time and then compare the performance. ...
- 164
12
votes
Why do we need to add START <s> + END </s> symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?
Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals ...
- 3,208
11
votes
Accepted
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
After a Googling around, I think this tutorial may suit your needs.
However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
- 23.8k
6
votes
How to determine feature importance in a neural network?
Linking to the same paper as @scholle but explaining the process differently (book and paper).
You do not need to train the model multiple times. The algorithm described in the links above require a ...
- 170
6
votes
Accepted
How do attention mechanisms in RNNs learn weights for a variable length input
Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size.
For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and ...
- 9,247
5
votes
How/What to initialize the hidden states in RNN sequence-to-sequence models?
It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural ...
- 95
5
votes
How are Q, K, and V Vectors Trained in a Transformer Self-Attention?
These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous ...
- 1,691
5
votes
Accepted
How does the Transformer predict n steps into the future?
The Transformer is a seq2seq model.
At training time, you pass to the Transformer model both the source and target tokens, just like what you do with LSTMs or GRUs with teacher forcing, which is the ...
- 23.8k
4
votes
Predict output sequence one at a time with feedback
I assume, that your data looks somehow like this (a pair of question and answer)
...
- 1,113
4
votes
ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)
The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here.
my original code is:
...
- 41
4
votes
How are Q, K, and V Vectors Trained in a Transformer Self-Attention?
Q, K, V vectors are trained with standard backpropagation. All trainable parameters are initialized at random, and then adjusted step by step with a Gradient Descent algorithm.
Surprisingly, they are ...
- 6,095
4
votes
Accepted
How to use BERT in seq2seq model?
In principle, it is possible to reuse the special tokens as you describe.
However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain ...
- 23.8k
3
votes
How to determine feature importance in a neural network?
You can do this sort of thing using SHAP, it looks at permutation importance as well.
3
votes
Methods for learning with noisy labels
First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples. In my opinion "noisy" is when positive and negative ...
- 25k
3
votes
Transformer seq2seq model and loading embeddings from XLM-RoBERTa
It is indeed possible, but the question is if it is a good idea.
FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and ...
- 1,691
3
votes
Accepted
Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder
About the need for tgt_key_padding_mask
While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before ...
- 23.8k
2
votes
Accepted
Training Encoder-Decoder using Decoder Outputs
I would like to know if it's possible to train a decoder by feeding
its predictions at time, t-1, into the input at time-step t.
Yes, it is possible to do it.
But I don't see why you would do it.
...
- 3,920
2
votes
Accepted
Do we really need <unk> tokens?
The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the ...
- 14.7k
2
votes
Very long sequence in neural networks
You can indeed use the ability of recurrent network like LSTM to handle the varying length problem.
But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a ...
- 1,083
2
votes
Can Sequence to sequence models be used to convert code from one programming language to another?
Yes, it is possible to convert code from one programming language to another using sequence-to-sequence neural networks. Sequence-to-sequence can learn to translate anything from anything if there are ...
- 20.5k
2
votes
Accepted
How can I feed BERT to neural machine translation?
I find this way of using BERT in my translation system and it allows me to load and use more data to train my model.
I got a memory error when I want to use more data like 100k for my task. and I came ...
- 41
2
votes
One-hot encode multi-class multi-label sequences
You may need to try cat2vec which converts categorical features into vector representation using Word2Vec approach. Check also this link for multi-feature inputs into LSTM.
For the target y, one-hot ...
2
votes
Sentences language translation with neural network, with a simple layer structure (if possible sequential)
Machine translation using traditional neural architecture (seq2seq models) had various issues due to rare-words, low accuracy and slow translation [1]. Even after using various mechanisms like ...
- 352
2
votes
Methods for learning with noisy labels
There is a python package created exactly for this purpose of finding label errors and training ML models robustly and reliably even when your data has issues or you have noisy labels: https://github....
- 151
2
votes
How are Q, K, and V Vectors Trained in a Transformer Self-Attention?
As Jindřich has said, Q, K, V come from previous computations, they are not trained directly with backpropagation. However, the weights $W_i^Q, W_i^K, W_i^V$ are trained directly with backpropagation.
...
2
votes
Why does an attention layer in a transformer learn context?
To provide a simplistic and less mathematical reasons. You can assume like this:
In a simple feed-forward neural network (a black-box of course), you shall learn the set of weights, learning a ...
- 1,059
2
votes
A simple attention based text prediction model from scratch using pytorch
Just for fun, run this for long generative output. Here is some code to put at the end. Also, you may want to change it to n-tuples or ngrams.
This is a nice toy language model!
...
2
votes
Multi-output, multi-timestep sequence prediction with Keras
To get an output on every step, you have to make return_sequence=True for all LSTM layers
The last Dense layer should reflect the output size i.e. 3 here
Hence, the ...
- 5,534
2
votes
Requirements for variable length output in transformer
The Transformer has the inherent ability to generate variable-length sequences, you don't need to do anything special. The output of the Transformer decoder is always the same length as the input of ...
- 23.8k
1
vote
SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors
It is, as you noticed, about the new version that changed the backend, now using something called eager execution.
The discussion on GitHub has some solutions though. Some suggested it is about ...
- 11
Only top scored, non community-wiki answers of a minimum length are eligible