12 votes
Accepted

How to determine feature importance in a neural network?

Don't remove a feature to find out its importance, but instead randomize or shuffle it. Run the training 10 times, randomize a different feature column each time and then compare the performance. ...
user avatar
  • 144
9 votes

Why do we need to add START <s> + END </s> symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals ...
user avatar
  • 3,028
6 votes
Accepted

How do attention mechanisms in RNNs learn weights for a variable length input

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and ...
user avatar
  • 8,629
5 votes
Accepted

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
user avatar
  • 16k
4 votes

Predict output sequence one at a time with feedback

I assume, that your data looks somehow like this (a pair of question and answer) ...
user avatar
4 votes

How/What to initialize the hidden states in RNN sequence-to-sequence models?

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural ...
user avatar
  • 85
4 votes

How to determine feature importance in a neural network?

Linking to the same paper as @scholle but explaining the process differently (book and paper). You do not need to train the model multiple times. The algorithm described in the links above require a ...
user avatar
4 votes

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous ...
user avatar
  • 1,406
4 votes
Accepted

How to use BERT in seq2seq model?

In principle, it is possible to reuse the special tokens as you describe. However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain ...
user avatar
  • 16k
3 votes

How to determine feature importance in a neural network?

You can do this sort of thing using SHAP, it looks at permutation importance as well.
user avatar
3 votes

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here. my original code is: ...
user avatar
3 votes

Methods for learning with noisy labels

First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples. In my opinion "noisy" is when positive and negative ...
user avatar
  • 22.1k
3 votes

Transformer seq2seq model and loading embeddings from XLM-RoBERTa

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and ...
user avatar
  • 1,406
3 votes
Accepted

Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder

About the need for tgt_key_padding_mask While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before ...
user avatar
  • 16k
2 votes
Accepted

Training Encoder-Decoder using Decoder Outputs

I would like to know if it's possible to train a decoder by feeding its predictions at time, t-1, into the input at time-step t. Yes, it is possible to do it. But I don't see why you would do it. ...
user avatar
  • 3,740
2 votes
Accepted

Do we really need <unk> tokens?

The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the ...
user avatar
  • 14k
2 votes

Very long sequence in neural networks

You can indeed use the ability of recurrent network like LSTM to handle the varying length problem. But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a ...
user avatar
  • 1,003
2 votes

Can Sequence to sequence models be used to convert code from one programming language to another?

Yes, it is possible to convert code from one programming language to another using sequence-to-sequence neural networks. Sequence-to-sequence can learn to translate anything from anything if there are ...
user avatar
2 votes
Accepted

How can I feed BERT to neural machine translation?

I find this way of using BERT in my translation system and it allows me to load and use more data to train my model. I got a memory error when I want to use more data like 100k for my task. and I came ...
user avatar
  • 41
2 votes

One-hot encode multi-class multi-label sequences

You may need to try cat2vec which converts categorical features into vector representation using Word2Vec approach. Check also this link for multi-feature inputs into LSTM. For the target y, one-hot ...
user avatar
2 votes

Sentences language translation with neural network, with a simple layer structure (if possible sequential)

Machine translation using traditional neural architecture (seq2seq models) had various issues due to rare-words, low accuracy and slow translation [1]. Even after using various mechanisms like ...
user avatar
2 votes

Why does an attention layer in a transformer learn context?

To provide a simplistic and less mathematical reasons. You can assume like this: In a simple feed-forward neural network (a black-box of course), you shall learn the set of weights, learning a ...
user avatar
2 votes
Accepted

How does the Transformer predict n steps into the future?

The Transformer is a seq2seq model. At training time, you pass to the Transformer model both the source and target tokens, just like what you do with LSTMs or GRUs with teacher forcing, which is the ...
user avatar
  • 16k
2 votes

Multi-output, multi-timestep sequence prediction with Keras

To get an output on every step, you have to make return_sequence=True for all LSTM layers The last Dense layer should reflect the output size i.e. 3 here Hence, the ...
user avatar
  • 5,184
1 vote

Answer to Question

Question answering (QA) is a complex problem and an active field of research. There are probably some academic prototypes around, but I doubt there's any general-purpose ready-to-use QA library. ...
user avatar
  • 22.1k
1 vote

SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors

It is, as you noticed, about the new version that changed the backend, now using something called eager execution. The discussion on GitHub has some solutions though. Some suggested it is about ...
user avatar
  • 11
1 vote

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

Luong's attention came after Bahdanau's and is generally considered an advancement over the former even though it has several simplifications. None of the pre-written layers I have seen, entirely ...
user avatar
  • 673
1 vote

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

In tensorflow-tutorials-for-text they are implementing bahdanau attention layer to generate context vector by giving encoder inputs, decoder hidden states and decoder inputs. Encoder class is ...
user avatar
1 vote

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

Here's the Deeplearning.ai notebook that is going to be helpful to understand it. Neural machine translation with attention
user avatar
  • 2,245
1 vote

How can I build a seq2seq model , which is topic aware

Try this approach (not embedded but filtering your model) with Topic modeling, that could be used to filter the results of your model by topic: Separate each doc in several sentences Obtain the best ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible