# Tag Info

11

Don't remove a feature to find out its importance, but instead randomize or shuffle it. Run the training 10 times, randomize a different feature column each time and then compare the performance. There is no need to tune hyper-parameters when done this way. Here's the theory behind my suggestion: feature importance

8

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. We (normally) don't care about intermediate states of the ...

4

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural networks. Since your question is asking about hidden state initialization: Hidden states on the other hand can be initialized in a variety of ways, initializing to ...

4

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and target sequence $\boldsymbol{y} = y_1...y_{T_y}$ (where $T_y$ can also vary from one target to another), weight \$\boldsymbol{\alpha}_i = (\alpha_{i1},\cdots,\...

4

I assume, that your data looks somehow like this (a pair of question and answer) Question,Answer How are you?,I'm fine. How are you?,OK. How are you?,Very well, thanks. I'd suggest to transform it in three features: Question (remains the same), InputSeq is the prefix of the answer generated in all possible lengths between zero and the number of words ...

4

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-based decoders are autoregressive by nature, Transformers are not. Instead, all predictions are generated at once based on the real target tokens (i.e. teacher ...

3

First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples. In my opinion "noisy" is when positive and negative cases are mixed together in a way that makes it difficult (or impossible) to distinguish between them. I think this matters because you're more likely to find ...

3

Linking to the same paper as @scholle but explaining the process differently (book and paper). You do not need to train the model multiple times. The algorithm described in the links above require a trained model to begin with. Given a trained model, compute the metric of interest on some dataset (the book discusses pros/cons of using training set vs test ...

3

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states. In Equation 1 of the ...

3

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and replace the encoder with XLM-R. Another option would be using Huggingface's Transofrmers that also provides basic support for sequence-to-sequence models as ...

3

In principle, it is possible to reuse the special tokens as you describe. However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain better translation quality. Another option would be to reuse just the embeddings instead of the whole model.

2

I would like to know if it's possible to train a decoder by feeding its predictions at time, t-1, into the input at time-step t. Yes, it is possible to do it. But I don't see why you would do it. Υou will have accumulated error propagated and amplified in every new prediction, making your prediction to diverge from the ground truth sooner or later.

2

The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the selection of a dictionary. If the word is not in the dictionary we have chosen, then we are saying we have no valid representation for that word (or we are simply not interested). Other tags are ...

2

You can indeed use the ability of recurrent network like LSTM to handle the varying length problem. But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a batch. What you can do : Pad all the sequences with an unused value (typically 0) so that all the sequences have the same length. Use a mask layer just after the ...

2

You can do this sort of thing using SHAP, it looks at permutation importance as well.

2

Yes, it is possible to convert code from one programming language to another using sequence-to-sequence neural networks. Sequence-to-sequence can learn to translate anything from anything if there are consistent patterns and enough paired examples. However, it would not be efficient because the models would only indirectly learn the semantics (i.e., the ...

2

You may need to try cat2vec which converts categorical features into vector representation using Word2Vec approach. Check also this link for multi-feature inputs into LSTM. For the target y, one-hot is a better technique for NN-based models.

2

The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here. my original code is: def sampling(args): z_mean, z_log_sigma = args epsilon = K.random_normal(shape=z_mean.shape) return z_mean + K.exp(z_log_sigma) * epsilon with this sampling method, I got the same error with yours. the ...

2

I find this way of using BERT in my translation system and it allows me to load and use more data to train my model. I got a memory error when I want to use more data like 100k for my task. and I came up that my tokenizer is a kind of problem here because it takes a lot of memory to make my tokenizer for such a huge volume of data so pre-trained models like ...

2

Machine translation using traditional neural architecture (seq2seq models) had various issues due to rare-words, low accuracy and slow translation [1]. Even after using various mechanisms like attention and residual connections the performance was only comparable (not better than) statistical phrase-based machine translations [1]. I can only think of this ...

2

To provide a simplistic and less mathematical reasons. You can assume like this: In a simple feed-forward neural network (a black-box of course), you shall learn the set of weights, learning a function to map inputs to outputs. But, in the transformers based architecture, you have Attentions. Here, the weights are structured into Query, Key and Value (Q,K,V)....

2

The Transformer is a seq2seq model. At training time, you pass to the Transformer model both the source and target tokens, just like what you do with LSTMs or GRUs with teacher forcing, which is the default way of training them. Note that, in the Transformer decoder, we need to apply masking to avoid the predictions depending on the current and future tokens....

2

About the need for tgt_key_padding_mask While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before normal tokens (i.e. left padding). For instance, fairseq supports parameter left_pad to specify precisely this. For left padding to be handled correctly, you must mask the padding tokens, because the ...

1

Question answering (QA) is a complex problem and an active field of research. There are probably some academic prototypes around, but I doubt there's any general-purpose ready-to-use QA library. However there are probably state of the art implementations for closed QA, i.e. QA restricted to a specific domain (I'm not aware of any specific library though). ...

1

It is, as you noticed, about the new version that changed the backend, now using something called eager execution. The discussion on GitHub has some solutions though. Some suggested it is about incompatibility between the layers. But as the title of the issue implies it is a pain in the neck. What may solve the problem is disabling it, which works in TF 2.1, ...

1

Luong's attention came after Bahdanau's and is generally considered an advancement over the former even though it has several simplifications. None of the pre-written layers I have seen, entirely implement Luong or Bahdanu's attention in entirety but only implement key pieces of those. It has been shown that major gains are seen in performance with the ...

1

In tensorflow-tutorials-for-text they are implementing bahdanau attention layer to generate context vector by giving encoder inputs, decoder hidden states and decoder inputs. Encoder class is simply passing the encoder inputs from Embedding layer to GRU layer along with encoder_states and returns encoder_outputs and ecoder_states. If we use LSTM instead ...

1

Here's the Deeplearning.ai notebook that is going to be helpful to understand it. Neural machine translation with attention

1

Try this approach (not embedded but filtering your model) with Topic modeling, that could be used to filter the results of your model by topic: Separate each doc in several sentences Obtain the best topics ( occur > 3, chars > 3 ) from sentences if you do not have an idea of topics to declare (LatentDirichlet Allocation) Discard some topics out of interest ...

1

It can be both. If you input the desired label and predict the next desired label, it's called teacher forcing. But using only this technique might hurt the performance at test time. So using the actual predicted output from the last time step is also a good idea. It's possible to do both : For each batch, with X% chance you use teacher forcing, ...

Only top voted, non community-wiki answers of a minimum length are eligible