Questions tagged [sequence-to-sequence]
The sequence-to-sequence tag has no usage guidance.
119
questions
0
votes
1
answer
305
views
Predict customer behaviour with Transformer(attention is all you need)
Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
0
votes
0
answers
28
views
How to reduce dimensionality of encoder decoder output?
I have an encoder decoder architecture where the output $ \bar{\bf{y}}_t $ is a sequence of integers of maximum length $n$. Each integer in the sequence is representative of a category so the ...
- 101
1
vote
1
answer
557
views
Why does an attention layer in a transformer learn context?
I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers.
What I'm confused on is why the ...
- 223
2
votes
1
answer
385
views
How to train a model on top of a transformer to output a sequence?
I am using huggingface to build a model that is capable of identifying mistakes in a given sentence.
Say I have a given sentence and a corresponding label as follows ->
...
- 529
2
votes
1
answer
32
views
Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)? [closed]
Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors?
If so, say for example we have a sentence and we pass it through the encoder to ...
- 23
1
vote
1
answer
1k
views
Pytorch LSTM not training
So I am currently trying to implement an LSTM on Pytorch, but for some reason the loss is not decreasing. Here is my network:
...
user98969
0
votes
1
answer
209
views
Why does the non autoregresive transfomer model in fairseq require the prev_output_tokens input?
fairseq includes an implementation of a non autoregressive transformer - which (as much as I understand) means that the whole output sequence is generated in a single forward run (in contrast to ...
- 101
3
votes
1
answer
194
views
When using padding in sequence models, is Keras validation accuracy valid/ reliable?
I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
- 173
2
votes
3
answers
140
views
What is the best way for synthetic data generation while maintaining privacy?
For one of the projects where we are working as third party contractors, we need a way for the company to share some datasets which can be used for data science. It is not possible for the company to ...
- 121
2
votes
0
answers
187
views
LSTM low training/validation error but really bad predictions
I'm building a LSTM model to create an automatic drums composer.
I'm following this post: LSTM Metallica
I've built my model and done all the enconding, I was able to emulate the behavior of the ...
1
vote
1
answer
399
views
How to add attention mechanism to my sequence-to-sequence architecture in Keras?
Based on this blog entry, I have written a sequence to sequence deep learning model in Keras:
...
- 987
1
vote
1
answer
146
views
Comparing Language Model of two corpora
I know using Conditional Language Model I can learn the probability of a sentence given the corpus I used to train my model. I will then be able to generate meaningful text by sampling from the ...
- 71
0
votes
1
answer
357
views
Feeding XLM-R embeddings to neural machine translation?
I’m very new to the field of deep learning. My aim is to make a translation between Catalan to Catalan Sign Language. The grammar of the two languages is different
Input: He sells food.
Output (...
- 31
1
vote
0
answers
126
views
Multiple time series sequence prediction for multiple multivariate time series
My question is somehow similar to this question, but not satisfied with the answer. I have 100 samples, each sample consists of ...
- 11
0
votes
1
answer
871
views
Transformer seq2seq model and loading embeddings from XLM-RoBERTa
Is it possible to feed embeddings from XLM- RoBERTa to transformer seq2seq model? I'm working on NMT that translates verbal language sentences to sign language sentences (e.g Input: He sells food. ...
- 31
1
vote
0
answers
41
views
Seq2Seq for sentence correction
I have a task in hand where I get a dirty formed sentence and need to correct it. Examples are, "StackOverflow best question answering platform" to be converted to "StackOverflow is best question ...
- 894
1
vote
1
answer
159
views
Sequence labeling with partially known labels
I am working on a sequence labeling task where, based on experience, many of the labels of a given input sequence can be reliably extracted with a simple rule-based approach. For example, considering ...
- 111
1
vote
2
answers
52
views
Which input to use when generating a new sequence
I want to use sequence-to-sequence architecture to generate sequences.
My data has such structure
...
1
vote
0
answers
195
views
How can I finetune XLM-R for neural machine translation between the same language(Catalan to Catalan-with different grammar structure))?
Can I fine-tune XLM-R to do something similar to https://www.guru99.com/seq2seq-model.html
(encoder-decoder architecture or any other way that perform good in small dataset - around 1000)
For example:
...
- 31
2
votes
1
answer
2k
views
Can I fine-tune BERT, ELMO or XLnet for Seq2Seq neural machine translation?
I'm working on neural machine translator that translates English sentences to American sign language sentences(e.g below). I've a quite small dataset - around 1000 sentence pairs. I'm wondering if it ...
- 31
0
votes
1
answer
2k
views
Is it possible feed BERT to seq2seq encoder/decoder NMT (for low resource language)?
I'm working on NMT model which the input and the target sentences are from the same language (but the grammar differs). I'm planning to pre-train and use BERT since I'm working on small dataset and ...
- 31
8
votes
3
answers
7k
views
How are Q, K, and V Vectors Trained in a Transformer Self-Attention?
I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
- 181
2
votes
2
answers
176
views
Methods for learning with noisy labels
I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels.
More precisely, I would like this method to be able to leverage noisy data as ...
- 166
1
vote
0
answers
34
views
How can I do a sequence to sequence model (RNN / LSTM) with Keras with fixed length data?
What I'm trying to do seems so simple, but I can't find any examples online. First, I'm not working in language, so all of the embedding stuff adds needless ...
- 65
4
votes
2
answers
244
views
Sentences language translation with neural network, with a simple layer structure (if possible sequential)
Context: Many language sentences translation systems (e.g. French to English) with neural networks use a seq2seq structure:
"the cat sat on the mat" -> [Seq2Seq ...
- 160
4
votes
1
answer
91
views
Answer to Question
Looking for a system which can generate answers to questions. Most systems and blogs posted on internet are on Question to answer but not on answer to question or paraphrasing or keyword to questions.
...
- 894
4
votes
1
answer
4k
views
SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors
I am writing Encoder-Decoder architecture with Bahdanau Attention using tf.keras with TensorFlow 2.0. Below is my code This is working with TensorFlow 1.15 but getting the error in 2.0. you can check ...
- 546
2
votes
1
answer
4k
views
How can I feed BERT to neural machine translation?
I am trying to feed the input and target sentences to an NMT model, I am trying to use BERT here, But I don't have any idea how ...
- 41
2
votes
0
answers
480
views
Weight matrices in transformers
I am trying to understand the transformer architecture.
I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
- 161
3
votes
3
answers
3k
views
Any good Implementations of Bi-LSTM bahdanau attention in Keras?
From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
- 347
1
vote
0
answers
131
views
Improving Performance of LSTM for time series prediction
My data consists of two features and a set of time series data labeled as "bookings". I have 1056 data point in the times series, for which I have two features for each. The data size is 1056x3.
My ...
- 85
0
votes
1
answer
67
views
two different attention methods for seq2seq
I see two different ways of applying attention in seq2seq:
(a) the context vector (the weighted sum of encoder hidden states) fed into the output softmax, as shown in the diagram below. The diagram ...
- 101
1
vote
0
answers
145
views
How to create a seq2seq without specifying a fixed decoder length?
Based on the model presented in this answer:
...
- 167
0
votes
1
answer
101
views
How can I build a seq2seq model , which is topic aware
I have developed a chatbot, which is basically a seq2seq LSTM network. Which can generate text based on input text. But the problem I am having right now is it is not topic aware.
As an example :
...
0
votes
1
answer
16
views
Confusion about Decoder labels for training seq-to-seq models
So in seq-to-seq models for say NMT, the decoder is a sequence model for the right-shifted intended output. My question is, during training, are the inputs and outputs of the decoder supposed to be ...
- 183
2
votes
0
answers
316
views
Why isnt my seq2seq model reconising what the <END> tag is?
I am making a seq2seq chatbot with the help of this guide: https://medium.com/predict/creating-a-chatbot-from-scratch-using-keras-and-tensorflow-59e8fc76be79 . I set up the model and data processing ...
- 31
0
votes
2
answers
118
views
Seq2Seq Model training: Encoder vs. Decoder
Can someone point me to an article which explains how the model training is done in Seq2Seq? I know "Teacher Forcing" is used but what I found so far hasn't been detailed enough. What I am most ...
- 223
2
votes
1
answer
4k
views
IndexError: list index out of range
I'm implementing a sequence-2-sequence model with RNN-VAE architecture, and I use an attention mechanism. I have problem in the decoder part.
I'm struggling with this error: IndexError: list index ...
- 614
5
votes
1
answer
7k
views
ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)
I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism.
...
- 614
0
votes
0
answers
34
views
Strategy for "forcing" number of labels in seq2seq predictions with Keras?
I'm trying to train a seq2seq model that for every timestep in a given timeseries sample will output 1 of 6 possible labels.
Furthermore, the training data is constructed in such a way that
Each ...
- 163
2
votes
1
answer
158
views
Working ofLSTM with multiple Units - NER
I am trying to understand working of LSTM networks and kind of not clear about how different neurons in a cell interact each other. I had a look at a similar question, but still not clear about few ...
- 123
1
vote
0
answers
37
views
Adding context in a sequence to sequence problem
The encoder of a seq2seq model is meant to generate a conditioning context for the decoder, as mentioned here
A RNN layer (or stack thereof) acts as "encoder": it processes the input sequence and ...
- 1,658
3
votes
1
answer
3k
views
One-hot encode multi-class multi-label sequences
Suppose I want to build a timeseries where each timestep is represented by a categorical array: the encoded sequences look like [[2, 0, 5],[3, 1, 4],..] and each ...
- 173
1
vote
0
answers
46
views
Initialising states in a multilayer sequence to sequence model
With a sequence to sequence model where the enocoder and decoder are both comprised of one layer each, the initial state of the decoder is initialised to use the final states of the encoder layer.
In ...
- 448
1
vote
0
answers
289
views
What should the size of the decoder output be in a sequence to sequence model
In a sequence to sequence model, a lot of the tutorials I have read state that the decoder target length should be the same as the encoder input length (https://blog.keras.io/building-autoencoders-in-...
- 448
18
votes
3
answers
39k
views
How to determine feature importance in a neural network?
I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the ...
- 448
2
votes
0
answers
1k
views
Recommended model for univariate or multivariate multistep ahead time series forecasting
I have a dataset consisting of recurring and non-recurring expense transactions from bank accounts, as well as other features describing the bank account and each transation. I aggregate these ...
- 189
3
votes
1
answer
4k
views
Predicting next number in a sequence - data analysis
I am a machine learning newbie and I am working on a project where I'm given a sequence of integers all of which are in the range 0 to 70. My goal is to predict the next integer in the sequence given ...
- 31
5
votes
2
answers
7k
views
Does this encoder-decoder LSTM make sense for time series sequence to sequence?
TASK
given $\vec x = [x_{t=-3}, x_{t=-2}, x_{t=-1}, x_{t=0}]$
predict $\vec y = [x_{t=1}, x_{t=2}]$
Whith an LSTM encoder-decoder (seq2seq)
MODEL
NOTE: the ? symbol in the shape of the tensors refers ...
- 1,658
4
votes
1
answer
2k
views
Why do position embeddings work?
In the papers "Convolutional Sequence to Sequence Learning" and
"Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order ...
- 1,317