Questions tagged [sequence-to-sequence]
The sequence-to-sequence tag has no usage guidance.
121
questions
18
votes
3
answers
43k
views
How to determine feature importance in a neural network?
I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the ...
- 458
15
votes
1
answer
12k
views
Why do we need to add START <s> + END </s> symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?
In the Sequence-to-Sequence models, we often see that the START (e.g. <s>) and END (e.g. </s>) symbols are added to ...
- 2,410
11
votes
2
answers
2k
views
How do attention mechanisms in RNNs learn weights for a variable length input
Attention mechanisms in RNNs are reasonably common to sequence to sequence models.
I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output ...
- 423
9
votes
2
answers
1k
views
Input for LSTM for financial time series directional prediction
I'm working on using an LSTM to predict the direction of the market for the next day.
My question concerns the input for the LSTM. My data is a financial time series $x_1 \ldots x_t$ where each $x_i$...
- 99
8
votes
4
answers
9k
views
How are Q, K, and V Vectors Trained in a Transformer Self-Attention?
I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
- 181
7
votes
1
answer
9k
views
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.).
For the purposes of this question, you may assume that I already have the input part ...
- 377
5
votes
1
answer
10k
views
How/What to initialize the hidden states in RNN sequence-to-sequence models?
In an RNN sequence-to-sequence model, the encode input hidden states and the output's hidden states needs to be initialized before training.
What values should we initialize them with? How should we ...
- 2,410
5
votes
1
answer
8k
views
ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)
I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism.
...
- 624
5
votes
1
answer
2k
views
Why do position embeddings work?
In the papers "Convolutional Sequence to Sequence Learning" and
"Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order ...
- 1,337
5
votes
2
answers
8k
views
Does this encoder-decoder LSTM make sense for time series sequence to sequence?
TASK
given $\vec x = [x_{t=-3}, x_{t=-2}, x_{t=-1}, x_{t=0}]$
predict $\vec y = [x_{t=1}, x_{t=2}]$
Whith an LSTM encoder-decoder (seq2seq)
MODEL
NOTE: the ? symbol in the shape of the tensors refers ...
- 1,668
4
votes
1
answer
2k
views
How does the Transformer predict n steps into the future?
I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...
- 43
4
votes
2
answers
254
views
Sentences language translation with neural network, with a simple layer structure (if possible sequential)
Context: Many language sentences translation systems (e.g. French to English) with neural networks use a seq2seq structure:
"the cat sat on the mat" -> [Seq2Seq ...
- 160
4
votes
1
answer
94
views
Answer to Question
Looking for a system which can generate answers to questions. Most systems and blogs posted on internet are on Question to answer but not on answer to question or paraphrasing or keyword to questions.
...
- 894
4
votes
1
answer
4k
views
SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors
I am writing Encoder-Decoder architecture with Bahdanau Attention using tf.keras with TensorFlow 2.0. Below is my code This is working with TensorFlow 1.15 but getting the error in 2.0. you can check ...
- 556
4
votes
2
answers
1k
views
Is this a problem for a Seq2Seq model?
I'm struggling to find a tutorial/example which covers using an seq2seq model for sequential inputs other then text/translation.
I have a multivariate dataset with n number of input variables each ...
- 93
3
votes
3
answers
147
views
Predict output sequence one at a time with feedback
I would like to solve the following classification problem: Given an input sequence and zero or more initial terms of the true output, predict the next term in the output sequence.
For example, my ...
- 2,381
3
votes
1
answer
3k
views
Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder
According to https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html, the forward function of nn.TransformerDecoder contemplates the following arguments:
tgt – the sequence to the ...
- 377
3
votes
3
answers
3k
views
Any good Implementations of Bi-LSTM bahdanau attention in Keras?
From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
- 347
3
votes
1
answer
2k
views
Multi-output, multi-timestep sequence prediction with Keras
I've been searching for about three hours and I can't find an answer to a very simple question.
I have a time series prediction problem. I am trying to use a Keras LSTM model (with a Dense at the end) ...
- 31
3
votes
1
answer
1k
views
A simple attention based text prediction model from scratch using pytorch
I first asked this question in codereview SE but a user recommended to post this here instead.
I have created a simple self attention based text prediction model using pytorch. The attention formula ...
- 301
3
votes
1
answer
3k
views
One-hot encode multi-class multi-label sequences
Suppose I want to build a timeseries where each timestep is represented by a categorical array: the encoded sequences look like [[2, 0, 5],[3, 1, 4],..] and each ...
- 173
3
votes
1
answer
223
views
When using padding in sequence models, is Keras validation accuracy valid/ reliable?
I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
- 173
3
votes
1
answer
60
views
String together a set of tokens into a sequence
I have this problem scenario - Given a set of tokens, string them or a subset of the tokens together using stop words into a sequence. I am clear that I can have potentially infinite pre-training data ...
- 141
3
votes
1
answer
4k
views
Predicting next number in a sequence - data analysis
I am a machine learning newbie and I am working on a project where I'm given a sequence of integers all of which are in the range 0 to 70. My goal is to predict the next integer in the sequence given ...
- 31
3
votes
0
answers
115
views
Encoder-Decoder Sequence-to-Sequence Model for Translations in Both Directions
Is it possible to use a pre-trained sequence to sequence encoder-decoder model which translates an input text in source language to an output in target language to do an inverse translation? That is, ...
- 193
2
votes
2
answers
193
views
Methods for learning with noisy labels
I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels.
More precisely, I would like this method to be able to leverage noisy data as ...
- 166
2
votes
1
answer
72
views
How to select the optimal beam size for beam search?
Most Text Generation Models use beam search to select the optimal output candidate. How does one choose the optimal beam size? It would probably vary from task to task, dataset to dataset, and model ...
2
votes
1
answer
4k
views
How can I feed BERT to neural machine translation?
I am trying to feed the input and target sentences to an NMT model, I am trying to use BERT here, But I don't have any idea how ...
- 41
2
votes
1
answer
388
views
Training Encoder-Decoder using Decoder Outputs
I am trying to build an encoder-decoder model for a text style transfer problem. The problem is I don't have parallel data between the two styles so I need to train the model in an unsupervised ...
- 217
2
votes
1
answer
2k
views
Very long sequence in neural networks
Beginner's question regarding sequences in neural networks: suppose I have classification problem that looks like:
X = very long sequence of varying length.
Y = class (assume for simplicity y=0/1).
...
- 31
2
votes
1
answer
321
views
Build a corpus for machine translation
I want to train an LSTM with attention for translation between French and a "rare" language. I say rare because it is an african language with less digital content, and especially databases ...
- 23
2
votes
1
answer
423
views
How to train a model on top of a transformer to output a sequence?
I am using huggingface to build a model that is capable of identifying mistakes in a given sentence.
Say I have a given sentence and a corresponding label as follows ->
...
- 529
2
votes
1
answer
34
views
Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)? [closed]
Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors?
If so, say for example we have a sentence and we pass it through the encoder to ...
- 23
2
votes
3
answers
150
views
What is the best way for synthetic data generation while maintaining privacy?
For one of the projects where we are working as third party contractors, we need a way for the company to share some datasets which can be used for data science. It is not possible for the company to ...
- 121
2
votes
1
answer
2k
views
Can I fine-tune BERT, ELMO or XLnet for Seq2Seq neural machine translation?
I'm working on neural machine translator that translates English sentences to American sign language sentences(e.g below). I've a quite small dataset - around 1000 sentence pairs. I'm wondering if it ...
- 31
2
votes
1
answer
161
views
Working ofLSTM with multiple Units - NER
I am trying to understand working of LSTM networks and kind of not clear about how different neurons in a cell interact each other. I had a look at a similar question, but still not clear about few ...
- 123
2
votes
2
answers
1k
views
Give Variable Length input to LSTM
My input data consist of list of list. Both list have dynamic length for every example like below.
...
- 226
2
votes
2
answers
306
views
What are the benefits and tradeoffs of a 1D conv vs a multi-input seq2seq LSTM model?
I have 6 sequences, s1,..,s6. Using all sequences I want to predict a binary vector q = [0,0,0,1,1,1,0,0,0,1,1,1,...], which is a mask of the activity of the 6 sequences.
I have looked at seq2seq ...
- 23
2
votes
0
answers
53
views
Attention mechanisms without a linear layer
I am currently looking into attention mechanism as they are used in (non-Transformer) encoder-decoder architectures, meaning an architecture where some RNN (usually LSTM or GRU) is used in both the ...
- 121
2
votes
0
answers
261
views
Why the LSTM on Keras does not work correctly when it is necessary to predict several steps forward
I used AirPassenger Dataset. And based on several previous values(for examples 20) I want to predict several(3 or 5) steps in future.
Like
X -> y
[10,20,30,....200]->[210,220,230]
[20,30,40,.......
2
votes
2
answers
531
views
Timeseries LSTM: does test data need to come after training data?
I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence).
Because the time series is very long, I plan ...
- 121
2
votes
0
answers
194
views
LSTM low training/validation error but really bad predictions
I'm building a LSTM model to create an automatic drums composer.
I'm following this post: LSTM Metallica
I've built my model and done all the enconding, I was able to emulate the behavior of the ...
2
votes
0
answers
522
views
Weight matrices in transformers
I am trying to understand the transformer architecture.
I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
- 161
2
votes
0
answers
324
views
Why isnt my seq2seq model reconising what the <END> tag is?
I am making a seq2seq chatbot with the help of this guide: https://medium.com/predict/creating-a-chatbot-from-scratch-using-keras-and-tensorflow-59e8fc76be79 . I set up the model and data processing ...
- 31
2
votes
1
answer
4k
views
IndexError: list index out of range
I'm implementing a sequence-2-sequence model with RNN-VAE architecture, and I use an attention mechanism. I have problem in the decoder part.
I'm struggling with this error: IndexError: list index ...
- 624
2
votes
0
answers
1k
views
Recommended model for univariate or multivariate multistep ahead time series forecasting
I have a dataset consisting of recurring and non-recurring expense transactions from bank accounts, as well as other features describing the bank account and each transation. I aggregate these ...
- 189
2
votes
1
answer
303
views
Keras: Softmax output into embedding layer
I'm trying to build an encoder-decoder network in Keras to generate a sentence of a particular style. As my problem is unsupervised i.e. I don't have the ground truths for the generated sentences, I ...
- 217
1
vote
1
answer
3k
views
How to use BERT in seq2seq model? [closed]
I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need ...
- 113
1
vote
1
answer
1k
views
Requirements for variable length output in transformer
I have been working on modifying the transformer from the article The Annotated Transformer. One of the features I would like to include is the ability to pass a sequence of fixed length, and receive ...
- 13
1
vote
1
answer
577
views
What happens when the length of input is shorter than length of output in transformer architecture?
Given standard transformer architecture with encoder and decoder.
What happens when the input for the encoder is shorter than the expected output from the decoder?
The decoder is expecting to receive ...
- 111