Questions tagged [sequence-to-sequence]
The sequence-to-sequence tag has no usage guidance.
86
questions
2
votes
1answer
42 views
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.).
For the purposes of this question, you may assume that I already have the input part ...
0
votes
0answers
19 views
Deciding on sequence prediction approach
I have a space of 128 states expressed with binary bitstrings. The transition probabilities among these states are known to me, as follows:
...
1
vote
0answers
30 views
Dummy Variables of Weights in RNN Backpropagation Through Time
In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that -
To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are defined to ...
0
votes
1answer
25 views
How does the Transformer predict n steps into the future?
I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...
0
votes
1answer
16 views
Difference between zero-padding and character-padding in Recurrent Neural Networks
For RNN's to work efficiently we vectorize the problem which results in an input matrix of shape
(m, max_seq_len)
where m is the number of examples, e.g. ...
1
vote
1answer
32 views
Timeseries LSTM: does test data need to come after training data?
I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence).
Because the time series is very long, I plan ...
0
votes
1answer
40 views
How to use BERT in seq2seq model? [closed]
I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need ...
0
votes
0answers
18 views
How far into the future can I forecast a time-series with an LSTM and strongly seasonal data
I am working on a Sequence-to-Sequence + Attention model for some time-series data. Now I have a really long time series, basically 40 years of daily observations for multiple sensors. The data itself ...
2
votes
1answer
25 views
Build a corpus for machine translation
I want to train an LSTM with attention for translation between French and a "rare" language. I say rare because it is an african language with less digital content, and especially databases ...
1
vote
1answer
24 views
Long range forecasting with sequence-to-sequence models
I have a task where I want to forecast daily observations for 1 year or 2 years in advance at multiple locations--so 365 or 730 days in advance. I actually have a pretty good dataset, meaning daily ...
0
votes
1answer
34 views
Predict customer behaviour with Transformer(attention is all you need)
Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
0
votes
0answers
23 views
How to reduce dimensionality of encoder decoder output?
I have an encoder decoder architecture where the output $ \bar{\bf{y}}_t $ is a sequence of integers of maximum length $n$. Each integer in the sequence is representative of a category so the ...
0
votes
0answers
10 views
Using Transcoder Model for language to language conversion
I have a problem statement like Converting deprecated code into a modern version of the same language. I'm currently converting with a custom Rule-based engine. But the modern version of the language ...
1
vote
1answer
61 views
Why does an attention layer in a transformer learn context?
I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers.
What I'm confused on is why the ...
0
votes
0answers
10 views
Learning conditional statements from natural language
Natural language text in emails might have conditional statements in them. Are there any technical papers and methods that explore converting unstructured text (eg. emails) into structured conditional ...
2
votes
1answer
45 views
How to train a model on top of a transformer to output a sequence?
I am using huggingface to build a model that is capable of identifying mistakes in a given sentence.
Say I have a given sentence and a corresponding label as follows ->
...
2
votes
1answer
23 views
Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)? [closed]
Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors?
If so, say for example we have a sentence and we pass it through the encoder to ...
0
votes
0answers
5 views
Case weights with sequence to sequence models?
I've got a bunch of variable-length sequences. They're basically vectors of zeros with the occasional one. Event rate is maybe 1%.
I want to build a model that will take sequences that I get in the ...
1
vote
1answer
82 views
Pytorch LSTM not training
So I am currently trying to implement an LSTM on Pytorch, but for some reason the loss is not decreasing. Here is my network:
...
0
votes
1answer
58 views
Why does the non autoregresive transfomer model in fairseq require the prev_output_tokens input?
fairseq includes an implementation of a non autoregressive transformer - which (as much as I understand) means that the whole output sequence is generated in a single forward run (in contrast to ...
0
votes
0answers
40 views
Sequence-to-Sequence Autoencoder with asymmetric length of encoding sequence
I am trying to design a sequence-to-sequence autoencoder where the encoding sequence has a length shorter than the sequence itself given that the encoding's dimensionality could be more to compensate ...
0
votes
0answers
20 views
How to approach a text parsing problem
I'm looking to parse human-generated cyber-physical sensor tags into a series of standardised tags and identifiers.
For example "Rm3.ZnT-SP" refers to ROOM ID:3 ZONE TEMPERATURE SETPOINT
The ...
1
vote
0answers
29 views
When using padding in sequence models, is Keras validation accuracy valid/ reliable?
I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
2
votes
3answers
69 views
What is the best way for synthetic data generation while maintaining privacy?
For one of the projects where we are working as third party contractors, we need a way for the company to share some datasets which can be used for data science. It is not possible for the company to ...
2
votes
0answers
48 views
LSTM low training/validation error but really bad predictions
I'm building a LSTM model to create an automatic drums composer.
I'm following this post: LSTM Metallica
I've built my model and done all the enconding, I was able to emulate the behavior of the ...
0
votes
0answers
243 views
Concatenating Encoder hidden states in LSTM pytorch
I am implementing a seq2seq autoencoder in pytorch:
Q1) While it is true that we can keep the encoder as bidirectional, but can we keep the decoder as bidirectional as well(does it make any sense) if ...
1
vote
1answer
65 views
How to add attention mechanism to my sequence-to-sequence architecture in Keras?
Based on this blog entry, I have written a sequence to sequence deep learning model in Keras:
...
1
vote
1answer
38 views
Comparing Language Model of two corpora
I know using Conditional Language Model I can learn the probability of a sentence given the corpus I used to train my model. I will then be able to generate meaningful text by sampling from the ...
0
votes
0answers
11 views
Tied embedding in Sequence to Sequence Task
Is it sensible to use a tied embedding between the encoder and decoder in a Sequence to Sequence task where the question and answering is within the same language?
This will lower the number of ...
0
votes
1answer
254 views
Feeding XLM-R embeddings to neural machine translation?
I’m very new to the field of deep learning. My aim is to make a translation between Catalan to Catalan Sign Language. The grammar of the two languages is different
Input: He sells food.
Output (...
1
vote
0answers
40 views
Multiple time series sequence prediction for multiple multivariate time series
My question is somehow similar to this question, but not satisfied with the answer. I have 100 samples, each sample consists of ...
0
votes
1answer
396 views
Transformer seq2seq model and loading embeddings from XLM-RoBERTa
Is it possible to feed embeddings from XLM- RoBERTa to transformer seq2seq model? I'm working on NMT that translates verbal language sentences to sign language sentences (e.g Input: He sells food. ...
1
vote
0answers
26 views
Seq2Seq for sentence correction
I have a task in hand where I get a dirty formed sentence and need to correct it. Examples are, "StackOverflow best question answering platform" to be converted to "StackOverflow is best question ...
1
vote
1answer
37 views
Sequence labeling with partially known labels
I am working on a sequence labeling task where, based on experience, many of the labels of a given input sequence can be reliably extracted with a simple rule-based approach. For example, considering ...
1
vote
2answers
30 views
Which input to use when generating a new sequence
I want to use sequence-to-sequence architecture to generate sequences.
My data has such structure
...
1
vote
0answers
140 views
How can I finetune XLM-R for neural machine translation between the same language(Catalan to Catalan-with different grammar structure))?
Can I fine-tune XLM-R to do something similar to https://www.guru99.com/seq2seq-model.html
(encoder-decoder architecture or any other way that perform good in small dataset - around 1000)
For example:
...
2
votes
1answer
1k views
Can I fine-tune BERT, ELMO or XLnet for Seq2Seq neural machine translation?
I'm working on neural machine translator that translates English sentences to American sign language sentences(e.g below). I've a quite small dataset - around 1000 sentence pairs. I'm wondering if it ...
0
votes
1answer
1k views
Is it possible feed BERT to seq2seq encoder/decoder NMT (for low resource language)?
I'm working on NMT model which the input and the target sentences are from the same language (but the grammar differs). I'm planning to pre-train and use BERT since I'm working on small dataset and ...
0
votes
3answers
851 views
How are Q, K, and V Vectors Trained in a Transformer Self-Attention?
I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
2
votes
1answer
53 views
Methods for learning with noisy labels
I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels.
More precisely, I would like this method to be able to leverage noisy data as ...
1
vote
0answers
21 views
How can I do a sequence to sequence model (RNN / LSTM) with Keras with fixed length data?
What I'm trying to do seems so simple, but I can't find any examples online. First, I'm not working in language, so all of the embedding stuff adds needless ...
3
votes
2answers
120 views
Sentences language translation with neural network, with a simple layer structure (if possible sequential)
Context: Many language sentences translation systems (e.g. French to English) with neural networks use a seq2seq structure:
"the cat sat on the mat" -> [Seq2Seq ...
3
votes
1answer
61 views
Answer to Question
Looking for a system which can generate answers to questions. Most systems and blogs posted on internet are on Question to answer but not on answer to question or paraphrasing or keyword to questions.
...
3
votes
1answer
3k views
SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors
I am writing Encoder-Decoder architecture with Bahdanau Attention using tf.keras with TensorFlow 2.0. Below is my code This is working with TensorFlow 1.15 but getting the error in 2.0. you can check ...
1
vote
1answer
3k views
How can I feed BERT to neural machine translation?
I am trying to feed the input and target sentences to an NMT model, I am trying to use BERT here, But I don't have any idea how ...
1
vote
0answers
157 views
Weight matrices in transformers
I am trying to understand the transformer architecture.
I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
2
votes
3answers
1k views
Any good Implementations of Bi-LSTM bahdanau attention in Keras?
From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
1
vote
0answers
36 views
Improving Performance of LSTM for time series prediction
My data consists of two features and a set of time series data labeled as "bookings". I have 1056 data point in the times series, for which I have two features for each. The data size is 1056x3.
My ...
0
votes
1answer
32 views
two different attention methods for seq2seq
I see two different ways of applying attention in seq2seq:
(a) the context vector (the weighted sum of encoder hidden states) fed into the output softmax, as shown in the diagram below. The diagram ...
1
vote
0answers
28 views
How to create a seq2seq without specifying a fixed decoder length?
Based on the model presented in this answer:
...
