# Tag Info

## Hot answers tagged sequence-to-sequence

9

Don't remove a feature to find out its importance, but instead randomize or shuffle it. Run the training 10 times, randomize a different feature column each time and then compare the performance. There is no need to tune hyper-parameters when done this way. Here's the theory behind my suggestion: feature importance

8

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. We (normally) don't care about intermediate states of the ...

4

I assume, that your data looks somehow like this (a pair of question and answer) Question,Answer How are you?,I'm fine. How are you?,OK. How are you?,Very well, thanks. I'd suggest to transform it in three features: Question (remains the same), InputSeq is the prefix of the answer generated in all possible lengths between zero and the number of words ...

4

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and target sequence $\boldsymbol{y} = y_1...y_{T_y}$ (where $T_y$ can also vary from one target to another), weight \$\boldsymbol{\alpha}_i = (\alpha_{i1},\cdots,\...

3

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural networks. Since your question is asking about hidden state initialization: Hidden states on the other hand can be initialized in a variety of ways, initializing to ...

3

First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples. In my opinion "noisy" is when positive and negative cases are mixed together in a way that makes it difficult (or impossible) to distinguish between them. I think this matters because you're more likely to find ...

3

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and replace the encoder with XLM-R. Another option would be using Huggingface's Transofrmers that also provides basic support for sequence-to-sequence models as ...

3

In principle, it is possible to reuse the special tokens as you describe. However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain better translation quality. Another option would be to reuse just the embeddings instead of the whole model.

3

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-based decoders are autoregressive by nature, Transformers are not. Instead, all predictions are generated at once based on the real target tokens (i.e. teacher ...

2

You can indeed use the ability of recurrent network like LSTM to handle the varying length problem. But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a batch. What you can do : Pad all the sequences with an unused value (typically 0) so that all the sequences have the same length. Use a mask layer just after the ...

2

I would like to know if it's possible to train a decoder by feeding its predictions at time, t-1, into the input at time-step t. Yes, it is possible to do it. But I don't see why you would do it. Υou will have accumulated error propagated and amplified in every new prediction, making your prediction to diverge from the ground truth sooner or later.

2

Yes, it is possible to convert code from one programming language to another using sequence-to-sequence neural networks. Sequence-to-sequence can learn to translate anything from anything if there are consistent patterns and enough paired examples. However, it would not be efficient because the models would only indirectly learn the semantics (i.e., the ...

2

I find this way of using BERT in my translation system and it allows me to load and use more data to train my model. I got a memory error when I want to use more data like 100k for my task. and I came up that my tokenizer is a kind of problem here because it takes a lot of memory to make my tokenizer for such a huge volume of data so pre-trained models like ...

2

You may need to try cat2vec which converts categorical features into vector representation using Word2Vec approach. Check also this link for multi-feature inputs into LSTM. For the target y, one-hot is a better technique for NN-based models.

2

Machine translation using traditional neural architecture (seq2seq models) had various issues due to rare-words, low accuracy and slow translation [1]. Even after using various mechanisms like attention and residual connections the performance was only comparable (not better than) statistical phrase-based machine translations [1]. I can only think of this ...

2

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states. In Equation 1 of the ...

2

To provide a simplistic and less mathematical reasons. You can assume like this: In a simple feed-forward neural network (a black-box of course), you shall learn the set of weights, learning a function to map inputs to outputs. But, in the transformers based architecture, you have Attentions. Here, the weights are structured into Query, Key and Value (Q,K,V)....

1

You can view models like ELMo or BERT to be encoder-only. They can be easily used for classification or sequence tagging, but the tag sequence is typically monotonically aligned with the source sequence. Even though the Transformer layers in BERT or XLNet are in theory capable of arbitrary reordering (which is used in non-autoregressive machine translation ...

1

Q, K, V vectors are trained with standard backpropagation. All trainable parameters are initialized at random, and then adjusted step by step with a Gradient Descent algorithm. Surprisingly, they are trained just as any standard ANN! It's pretty amazing what they can achieve with such a classical trick.

1

Question answering (QA) is a complex problem and an active field of research. There are probably some academic prototypes around, but I doubt there's any general-purpose ready-to-use QA library. However there are probably state of the art implementations for closed QA, i.e. QA restricted to a specific domain (I'm not aware of any specific library though). ...

1

You can do this sort of thing using SHAP, it looks at permutation importance as well.

1

Yes, it makes sense. Seq2seq models represent, in the RNN family, the best for multistep predictions. More classical RNNs, on the other side, are not that good for predicting long sequences. If you need to implement a seq2seq model in TensorFlow 2.0 / Keras, each model follows the following structure: from tensorflow.keras.models import Sequential from ...

1

You can try padding the inputs with zero that is of lesser in length.

1

In short your friend is correct, seq2seq is a reasonable match to the problem. However, from your numbers this could be too complex to use current machine learning libraries on. Despite you calling it "not feasible", you are far better off with the reverse engineering effort in my opinion. If you really want to have a try using ML, you could start with ...

1

Luong's attention came after Bahdanau's and is generally considered an advancement over the former even though it has several simplifications. None of the pre-written layers I have seen, entirely implement Luong or Bahdanu's attention in entirety but only implement key pieces of those. It has been shown that major gains are seen in performance with the ...

1

In tensorflow-tutorials-for-text they are implementing bahdanau attention layer to generate context vector by giving encoder inputs, decoder hidden states and decoder inputs. Encoder class is simply passing the encoder inputs from Embedding layer to GRU layer along with encoder_states and returns encoder_outputs and ecoder_states. If we use LSTM instead ...

1

Here's the Deeplearning.ai notebook that is going to be helpful to understand it. Neural machine translation with attention

1

Try this approach (not embedded but filtering your model) with Topic modeling, that could be used to filter the results of your model by topic: Separate each doc in several sentences Obtain the best topics ( occur > 3, chars > 3 ) from sentences if you do not have an idea of topics to declare (LatentDirichlet Allocation) Discard some topics out of interest ...

1

It can be both. If you input the desired label and predict the next desired label, it's called teacher forcing. But using only this technique might hurt the performance at test time. So using the actual predicted output from the last time step is also a good idea. It's possible to do both : For each batch, with X% chance you use teacher forcing, ...

1

The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here. my original code is: def sampling(args): z_mean, z_log_sigma = args epsilon = K.random_normal(shape=z_mean.shape) return z_mean + K.exp(z_log_sigma) * epsilon with this sampling method, I got the same error with yours. the ...

Only top voted, non community-wiki answers of a minimum length are eligible