7

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. We (normally) don't care about intermediate states of the ...


6

Don't remove a feature to find out its importance, but instead randomize or shuffle it. Run the training 10 times, randomize a different feature column each time and then compare the performance. There is no need to tune hyper-parameters when done this way. Here's the theory behind my suggestion: feature importance


4

I assume, that your data looks somehow like this (a pair of question and answer) Question,Answer How are you?,I'm fine. How are you?,OK. How are you?,Very well, thanks. I'd suggest to transform it in three features: Question (remains the same), InputSeq is the prefix of the answer generated in all possible lengths between zero and the number of words ...


3

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and replace the encoder with XLM-R. Another option would be using Huggingface's Transofrmers that also provides basic support for sequence-to-sequence models as ...


3

First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples. In my opinion "noisy" is when positive and negative cases are mixed together in a way that makes it difficult (or impossible) to distinguish between them. I think this matters because you're more likely to find ...


3

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and target sequence $\boldsymbol{y} = y_1...y_{T_y}$ (where $T_y$ can also vary from one target to another), weight $\boldsymbol{\alpha}_i = (\alpha_{i1},\cdots,\...


2

You can indeed use the ability of recurrent network like LSTM to handle the varying length problem. But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a batch. What you can do : Pad all the sequences with an unused value (typically 0) so that all the sequences have the same length. Use a mask layer just after the ...


2

I would like to know if it's possible to train a decoder by feeding its predictions at time, t-1, into the input at time-step t. Yes, it is possible to do it. But I don't see why you would do it. Υou will have accumulated error propagated and amplified in every new prediction, making your prediction to diverge from the ground truth sooner or later.


2

Yes, it is possible to convert code from one programming language to another using sequence-to-sequence neural networks. Sequence-to-sequence can learn to translate anything from anything if there are consistent patterns and enough paired examples. However, it would not be efficient because the models would only indirectly learn the semantics (i.e., the ...


2

Machine translation using traditional neural architecture (seq2seq models) had various issues due to rare-words, low accuracy and slow translation [1]. Even after using various mechanisms like attention and residual connections the performance was only comparable (not better than) statistical phrase-based machine translations [1]. I can only think of this ...


2

You may need to try cat2vec which converts categorical features into vector representation using Word2Vec approach. Check also this link for multi-feature inputs into LSTM. For the target y, one-hot is a better technique for NN-based models.


2

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural networks. Since your question is asking about hidden state initialization: Hidden states on the other hand can be initialized in a variety of ways, initializing to ...


1

If I understand your question correctly (please comment if I don't), you want to compare the output text from two different language models. Therefore, I don't think you should worry too much about the language models themselves. If I was you, I'd probably do a simple TFIDF analysis so that you can gain a better understanding of what terms are more ...


1

You can train a regular sequence labeling model (typically CRF) where one of the features is the rule-based predicted label: its value is the actual label when known or a special unknown value otherwise. Given that the model can take into account dependencies between labels (as specified in the parameters) and that the rule-based feature always gives the ...


1

You can still use 0 as Start Of the Sequence token. Shift the input data by adding a constant to all values, for example adding 10. Then prepend a 0 to the input. A linear transformation of input will not affect the ability of machine learning models to learn. Make sure to apply the same transformation to both training and prediction stages.


1

As you have seen, normally you need a "special token" to be given to the decoder as the first element in its input to start the autoregressive generation. However, given that your output are real (floating point) numbers, it is a bit trickier, as you are not dealing with a discrete token vocabulary where you could simply reserve a token for that. I would ...


1

You can view models like ELMo or BERT to be encoder-only. They can be easily used for classification or sequence tagging, but the tag sequence is typically monotonically aligned with the source sequence. Even though the Transformer layers in BERT or XLNet are in theory capable of arbitrary reordering (which is used in non-autoregressive machine translation ...


1

Q, K, V vectors are trained with standard backpropagation. All trainable parameters are initialized at random, and then adjusted step by step with a Gradient Descent algorithm. Surprisingly, they are trained just as any standard ANN! It's pretty amazing what they can achieve with such a classical trick.


1

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states. In Equation 1 of the ...


1

I find this way of using BERT in my translation system and it allows me to load and use more data to train my model. I got a memory error when I want to use more data like 100k for my task. and I came up that my tokenizer is a kind of problem here because it takes a lot of memory to make my tokenizer for such a huge volume of data so pre-trained models like ...


1

Here's the Deeplearning.ai notebook that is going to be helpful to understand it. Neural machine translation with attention


1

Try this approach (not embedded but filtering your model) with Topic modeling, that could be used to filter the results of your model by topic: Separate each doc in several sentences Obtain the best topics ( occur > 3, chars > 3 ) from sentences if you do not have an idea of topics to declare (LatentDirichlet Allocation) Discard some topics out of interest ...


1

It can be both. If you input the desired label and predict the next desired label, it's called teacher forcing. But using only this technique might hurt the performance at test time. So using the actual predicted output from the last time step is also a good idea. It's possible to do both : For each batch, with X% chance you use teacher forcing, ...


1

Welcome to StackExchange DataScience! 1) The same input $x_1$ and $h_0$ will be fed to each 100 neurons parallel (independent) and respective outputs will go to next time step. Notice that there are two 100-dimensional vector going to the next time step: $c_1$ and $h_1$, which are denoted by the arrows on the right side in your second figure. 2) This is a ...


1

Yes, it makes sense. Seq2seq models represent, in the RNN family, the best for multistep predictions. More classical RNNs, on the other side, are not that good for predicting long sequences. If you need to implement a seq2seq model in TensorFlow 2.0 / Keras, each model follows the following structure: from tensorflow.keras.models import Sequential from ...


1

You can try padding the inputs with zero that is of lesser in length.


1

In short your friend is correct, seq2seq is a reasonable match to the problem. However, from your numbers this could be too complex to use current machine learning libraries on. Despite you calling it "not feasible", you are far better off with the reverse engineering effort in my opinion. If you really want to have a try using ML, you could start with ...


1

After some more research, I believe this can be solved with a straightforward application of encoder-decoder networks. In this tutorial, we can simply replace sampled_token_index and sampled_char with actual_token_index and actual_char, computed according. And of course in our case it's actual_word. To summarize we divide our training set into input/output ...


1

The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the selection of a dictionary. If the word is not in the dictionary we have chosen, then we are saying we have no valid representation for that word (or we are simply not interested). Other tags are ...


1

I see your problem consisting of two parts: Predicting which users will participate. Predicting the number of groups and what individual group composition will be. For the first part, you can use Hidden Markov Models (HMMs) for each user id which basically model the probabilities such that given the prior knowledge of user participation ('PNNPPPN') it ...


Only top voted, non community-wiki answers of a minimum length are eligible