Skip to main content

Questions tagged [sequence-to-sequence]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
18 votes
3 answers
44k views

How to determine feature importance in a neural network?

I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the ...
Aesir's user avatar
  • 458
15 votes
1 answer
12k views

Why do we need to add START <s> + END </s> symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?

In the Sequence-to-Sequence models, we often see that the START (e.g. <s>) and END (e.g. </s>) symbols are added to ...
alvas's user avatar
  • 2,410
11 votes
2 answers
2k views

How do attention mechanisms in RNNs learn weights for a variable length input

Attention mechanisms in RNNs are reasonably common to sequence to sequence models. I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output ...
davidparks21's user avatar
9 votes
2 answers
1k views

Input for LSTM for financial time series directional prediction

I'm working on using an LSTM to predict the direction of the market for the next day. My question concerns the input for the LSTM. My data is a financial time series $x_1 \ldots x_t$ where each $x_i$...
articuno's user avatar
8 votes
4 answers
10k views

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
arctic_hen7's user avatar
7 votes
1 answer
10k views

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.). For the purposes of this question, you may assume that I already have the input part ...
Pablo Messina's user avatar
5 votes
1 answer
10k views

How/What to initialize the hidden states in RNN sequence-to-sequence models?

In an RNN sequence-to-sequence model, the encode input hidden states and the output's hidden states needs to be initialized before training. What values should we initialize them with? How should we ...
alvas's user avatar
  • 2,410
5 votes
1 answer
8k views

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism. ...
Kahina's user avatar
  • 634
5 votes
1 answer
2k views

Why do position embeddings work?

In the papers "Convolutional Sequence to Sequence Learning" and "Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order ...
Robin's user avatar
  • 1,337
5 votes
2 answers
8k views

Does this encoder-decoder LSTM make sense for time series sequence to sequence?

TASK given $\vec x = [x_{t=-3}, x_{t=-2}, x_{t=-1}, x_{t=0}]$ predict $\vec y = [x_{t=1}, x_{t=2}]$ Whith an LSTM encoder-decoder (seq2seq) MODEL NOTE: the ? symbol in the shape of the tensors refers ...
ignatius's user avatar
  • 1,668
4 votes
1 answer
2k views

How does the Transformer predict n steps into the future?

I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...
skevelis's user avatar
4 votes
2 answers
259 views

Sentences language translation with neural network, with a simple layer structure (if possible sequential)

Context: Many language sentences translation systems (e.g. French to English) with neural networks use a seq2seq structure: "the cat sat on the mat" -> [Seq2Seq ...
Basj's user avatar
  • 160
4 votes
1 answer
94 views

Answer to Question

Looking for a system which can generate answers to questions. Most systems and blogs posted on internet are on Question to answer but not on answer to question or paraphrasing or keyword to questions. ...
Sandeep Bhutani's user avatar
4 votes
1 answer
4k views

SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors

I am writing Encoder-Decoder architecture with Bahdanau Attention using tf.keras with TensorFlow 2.0. Below is my code This is working with TensorFlow 1.15 but getting the error in 2.0. you can check ...
Uday's user avatar
  • 556
4 votes
2 answers
1k views

Is this a problem for a Seq2Seq model?

I'm struggling to find a tutorial/example which covers using an seq2seq model for sequential inputs other then text/translation. I have a multivariate dataset with n number of input variables each ...
Ellio's user avatar
  • 93
3 votes
3 answers
148 views

Predict output sequence one at a time with feedback

I would like to solve the following classification problem: Given an input sequence and zero or more initial terms of the true output, predict the next term in the output sequence. For example, my ...
Imran's user avatar
  • 2,381
3 votes
1 answer
3k views

Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder

According to https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html, the forward function of nn.TransformerDecoder contemplates the following arguments: tgt – the sequence to the ...
Pablo Messina's user avatar
3 votes
3 answers
3k views

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
user_12's user avatar
  • 347
3 votes
1 answer
2k views

Multi-output, multi-timestep sequence prediction with Keras

I've been searching for about three hours and I can't find an answer to a very simple question. I have a time series prediction problem. I am trying to use a Keras LSTM model (with a Dense at the end) ...
MGreen's user avatar
  • 31
3 votes
1 answer
1k views

A simple attention based text prediction model from scratch using pytorch

I first asked this question in codereview SE but a user recommended to post this here instead. I have created a simple self attention based text prediction model using pytorch. The attention formula ...
Eka's user avatar
  • 301
3 votes
1 answer
3k views

One-hot encode multi-class multi-label sequences

Suppose I want to build a timeseries where each timestep is represented by a categorical array: the encoded sequences look like [[2, 0, 5],[3, 1, 4],..] and each ...
ginevracoal's user avatar
3 votes
1 answer
228 views

When using padding in sequence models, is Keras validation accuracy valid/ reliable?

I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
Amir Jalilifard's user avatar
3 votes
1 answer
60 views

String together a set of tokens into a sequence

I have this problem scenario - Given a set of tokens, string them or a subset of the tokens together using stop words into a sequence. I am clear that I can have potentially infinite pre-training data ...
Deepak Saini's user avatar
3 votes
1 answer
4k views

Predicting next number in a sequence - data analysis

I am a machine learning newbie and I am working on a project where I'm given a sequence of integers all of which are in the range 0 to 70. My goal is to predict the next integer in the sequence given ...
varun's user avatar
  • 31
3 votes
0 answers
115 views

Encoder-Decoder Sequence-to-Sequence Model for Translations in Both Directions

Is it possible to use a pre-trained sequence to sequence encoder-decoder model which translates an input text in source language to an output in target language to do an inverse translation? That is, ...
Amir's user avatar
  • 193
2 votes
2 answers
197 views

Methods for learning with noisy labels

I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels. More precisely, I would like this method to be able to leverage noisy data as ...
Mathias Müller's user avatar
2 votes
1 answer
194 views

How to select the optimal beam size for beam search?

Most Text Generation Models use beam search to select the optimal output candidate. How does one choose the optimal beam size? It would probably vary from task to task, dataset to dataset, and model ...
Tathagato Roy's user avatar
2 votes
2 answers
601 views

Timeseries LSTM: does test data need to come after training data?

I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence). Because the time series is very long, I plan ...
lodo's user avatar
  • 121
2 votes
1 answer
4k views

How can I feed BERT to neural machine translation?

I am trying to feed the input and target sentences to an NMT model, I am trying to use BERT here, But I don't have any idea how ...
Hamed's user avatar
  • 39
2 votes
1 answer
389 views

Training Encoder-Decoder using Decoder Outputs

I am trying to build an encoder-decoder model for a text style transfer problem. The problem is I don't have parallel data between the two styles so I need to train the model in an unsupervised ...
Physbox's user avatar
  • 217
2 votes
1 answer
2k views

Very long sequence in neural networks

Beginner's question regarding sequences in neural networks: suppose I have classification problem that looks like: X = very long sequence of varying length. Y = class (assume for simplicity y=0/1). ...
Roger T's user avatar
  • 31
2 votes
1 answer
343 views

Build a corpus for machine translation

I want to train an LSTM with attention for translation between French and a "rare" language. I say rare because it is an african language with less digital content, and especially databases ...
Meomeoowww's user avatar
2 votes
1 answer
439 views

How to train a model on top of a transformer to output a sequence?

I am using huggingface to build a model that is capable of identifying mistakes in a given sentence. Say I have a given sentence and a corresponding label as follows -> ...
MetaInformation's user avatar
2 votes
1 answer
35 views

Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)? [closed]

Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors? If so, say for example we have a sentence and we pass it through the encoder to ...
Dhruv's user avatar
  • 23
2 votes
3 answers
151 views

What is the best way for synthetic data generation while maintaining privacy?

For one of the projects where we are working as third party contractors, we need a way for the company to share some datasets which can be used for data science. It is not possible for the company to ...
Vivek Maskara's user avatar
2 votes
1 answer
2k views

Can I fine-tune BERT, ELMO or XLnet for Seq2Seq neural machine translation?

I'm working on neural machine translator that translates English sentences to American sign language sentences(e.g below). I've a quite small dataset - around 1000 sentence pairs. I'm wondering if it ...
NLP Dude's user avatar
2 votes
1 answer
164 views

Working ofLSTM with multiple Units - NER

I am trying to understand working of LSTM networks and kind of not clear about how different neurons in a cell interact each other. I had a look at a similar question, but still not clear about few ...
bkuriach's user avatar
  • 123
2 votes
2 answers
1k views

Give Variable Length input to LSTM

My input data consist of list of list. Both list have dynamic length for every example like below. ...
Batuhan B's user avatar
  • 226
2 votes
2 answers
309 views

What are the benefits and tradeoffs of a 1D conv vs a multi-input seq2seq LSTM model?

I have 6 sequences, s1,..,s6. Using all sequences I want to predict a binary vector q = [0,0,0,1,1,1,0,0,0,1,1,1,...], which is a mask of the activity of the 6 sequences. I have looked at seq2seq ...
Zach LeFevre's user avatar
2 votes
0 answers
65 views

Attention mechanisms without a linear layer

I am currently looking into attention mechanism as they are used in (non-Transformer) encoder-decoder architectures, meaning an architecture where some RNN (usually LSTM or GRU) is used in both the ...
krise's user avatar
  • 121
2 votes
0 answers
264 views

Why the LSTM on Keras does not work correctly when it is necessary to predict several steps forward

I used AirPassenger Dataset. And based on several previous values(for examples 20) I want to predict several(3 or 5) steps in future. Like X -> y [10,20,30,....200]->[210,220,230] [20,30,40,.......
Vladimir Shebuniayeu's user avatar
2 votes
0 answers
196 views

LSTM low training/validation error but really bad predictions

I'm building a LSTM model to create an automatic drums composer. I'm following this post: LSTM Metallica I've built my model and done all the enconding, I was able to emulate the behavior of the ...
Mattia Surricchio's user avatar
2 votes
0 answers
529 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
boredaf's user avatar
  • 161
2 votes
0 answers
326 views

Why isnt my seq2seq model reconising what the <END> tag is?

I am making a seq2seq chatbot with the help of this guide: https://medium.com/predict/creating-a-chatbot-from-scratch-using-keras-and-tensorflow-59e8fc76be79 . I set up the model and data processing ...
Dup Dup's user avatar
  • 31
2 votes
1 answer
4k views

IndexError: list index out of range

I'm implementing a sequence-2-sequence model with RNN-VAE architecture, and I use an attention mechanism. I have problem in the decoder part. I'm struggling with this error: IndexError: list index ...
Kahina's user avatar
  • 634
2 votes
0 answers
1k views

Recommended model for univariate or multivariate multistep ahead time series forecasting

I have a dataset consisting of recurring and non-recurring expense transactions from bank accounts, as well as other features describing the bank account and each transation. I aggregate these ...
KOB's user avatar
  • 189
2 votes
1 answer
309 views

Keras: Softmax output into embedding layer

I'm trying to build an encoder-decoder network in Keras to generate a sentence of a particular style. As my problem is unsupervised i.e. I don't have the ground truths for the generated sentences, I ...
Physbox's user avatar
  • 217
1 vote
1 answer
3k views

How to use BERT in seq2seq model? [closed]

I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need ...
Andrey's user avatar
  • 113
1 vote
1 answer
2k views

Requirements for variable length output in transformer

I have been working on modifying the transformer from the article The Annotated Transformer. One of the features I would like to include is the ability to pass a sequence of fixed length, and receive ...
try_hard's user avatar
1 vote
1 answer
619 views

What happens when the length of input is shorter than length of output in transformer architecture?

Given standard transformer architecture with encoder and decoder. What happens when the input for the encoder is shorter than the expected output from the decoder? The decoder is expecting to receive ...
Damian Grzanka's user avatar