Questions tagged [sequence-to-sequence]
The sequence-to-sequence tag has no usage guidance.
104
questions
1
vote
0answers
6 views
Composite Input into Seq2Seq LSTM Network
Given that we have a seq2seq problem, where the input sequence is indeed multiple inputs and not only one as in traditional seq2seq problems. For example, in language translation, we usually give ...
0
votes
0answers
16 views
What is the SOTA in Pointer Networks?
I have more or less a sequence2sequence generation task. The special nature is that the correct answer is guaranteed to be a combination of the tokens in the input. ...
0
votes
0answers
9 views
Autoencoder in keras and accuracy
I am looking at Autoencoders in keras. They say,
"Autoencoding" is a data compression algorithm where the compression
and decompression functions are 1) data-specific, 2) lossy, and 3)
...
0
votes
1answer
12 views
Why is sequence prediction always the objective in RNN and LSTM like algorithms
The title is pretty much my question. I haven't seen any literature yet that uses a different training objective. The goal is to find the hidden states eventually, then why is it that only 1 method is ...
2
votes
0answers
22 views
Why the LSTM on Keras does not work correctly when it is necessary to predict several steps forward
I used AirPassenger Dataset. And based on several previous values(for examples 20) I want to predict several(3 or 5) steps in future.
Like
X -> y
[10,20,30,....200]->[210,220,230]
[20,30,40,.......
0
votes
0answers
12 views
Sequence to Sequence learning applied to list of numbers
I am looking to apply ML methods to genetic data. My goal is to predict which rare (generally de novo) mutations a person has based on what non-rare (generally inherited) mutations.
I have worked on ...
0
votes
0answers
20 views
Empty prediction with keras Seq2Seq with attention mechanism
I have a simply seq2seq model with attention mechanism in keras. My problem is that the inference model only gives me empty prediction. However, if I remove the attention it suvessfully gives me the ...
0
votes
1answer
12 views
In keras seq2seq model, what is the difference between `model.predict()` and the inference model?
I am looking into seq2seq model in keras, for example, this blog post from keras or this. All the examples I have seen have some inference model, that depicts the original model. That inference model ...
1
vote
0answers
10 views
Sequence learning from farm operations data
I need to generalize a single sequence from N sequences entailing farming tasks/operations and ultimately plotting it on Gantt chart.
There are a total let's assume N sequences = n (total fields) * t (...
1
vote
1answer
75 views
Multi-step forecasts of factory production data using a Seq2Seq Encoder-Decoder Model with Attention
I am attempting to use a Seq2Seq model to make forecasts of factory production data using an Encoder-Decoder model augmented with Attention. I have become a little stuck as the output of the model ...
0
votes
1answer
36 views
Attention weights - change during learning and prediction
Assume a simple LSTM Followed by Attention layer or a full transformer architecture. The attention weights are learnt during training, which get multiplied with keys, queries and values.
Please ...
0
votes
1answer
29 views
Preprocess multi-sample time series data: encode each sample separately or in aggregate?
Let's say I have 3 dense sequences of uniform length. Should I fit a scaler on them separately or together?
...
0
votes
0answers
21 views
Finding recurring patterns in a (non-numeric) sequence
Suppose we have a long sequence of events (of the form ABCBBBNFABCBNF...ABC), and we want to detect:
Exact subsequences above a certain length, and which recur above a certain number of times (e.g. ...
1
vote
0answers
10 views
Workaround / fallback value for tfp.distributions.Categorical.log_prob in tensorflow graph mode
Is there a way to avoid tfp.distributions.Categorical.log_probraising an error if the input is a label out of range?
I am passing a batch of samples to the ...
0
votes
0answers
58 views
ValueError: Input 0 of layer conv_lst_m2d_60 is incompatible with the layer: expected ndim=5, found ndim=4. Full shape received: (None, 7, 7, 512)
I am building an anomaly detection model using keras upon videos. There are total 179 frames. The original dimension of each frame is given below:
...
0
votes
0answers
23 views
How to use gradient checkpointing on packed sequence RNN
I have a batch of sequences that have a variable length. To save computation I used pack_padded_sequence as following:
...
1
vote
1answer
259 views
Calculating confidence score in NER
I am working on a problem on Named Entity Recognition. Given a text, my model is detecting the Named Entities and extracting that info for the end-user. Now the ask is end-user needs a confidence ...
2
votes
1answer
117 views
Multi-output, multi-timestep sequence prediction with Keras
I've been searching for about three hours and I can't find an answer to a very simple question.
I have a time series prediction problem. I am trying to use a Keras LSTM model (with a Dense at the end) ...
1
vote
1answer
52 views
A simple attention based text prediction model from scratch using pytorch
I first asked this question in codereview SE but a user recommended to post this here instead.
I have created a simple self attention based text prediction model using pytorch. The attention formula ...
2
votes
1answer
28 views
How to implement sequence to sequence models?
I have a dataset with patient demographics, diagnosis history, hospital visit dates, drugs consumed etc.
All these events have time stamp information (except static info like demographics such gender, ...
0
votes
0answers
21 views
How to compute the loss of seq2seq model's variable length output sequence against the target sequence?
To train a seq2seq model without teacher forcing, we will use the decoder to generate a sequence, compare it against the target sequence, and compute the loss. One such example can be found here:
But ...
0
votes
1answer
184 views
Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder
According to https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html, the forward function of nn.TransformerDecoder contemplates the following arguments:
tgt – the sequence to the ...
0
votes
1answer
13 views
Different training method for encoder-decoder model
Trying to learn the encoder-decoder model for some NLP problems.
I am referring to this Keras tutorial.
During the model training phase, this tutorial just uses the following:
...
3
votes
1answer
261 views
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.).
For the purposes of this question, you may assume that I already have the input part ...
1
vote
0answers
36 views
Dummy Variables of Weights in RNN Backpropagation Through Time
In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that -
To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are defined to ...
0
votes
1answer
57 views
How does the Transformer predict n steps into the future?
I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...
0
votes
1answer
26 views
Difference between zero-padding and character-padding in Recurrent Neural Networks
For RNN's to work efficiently we vectorize the problem which results in an input matrix of shape
(m, max_seq_len)
where m is the number of examples, e.g. ...
1
vote
1answer
78 views
Timeseries LSTM: does test data need to come after training data?
I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence).
Because the time series is very long, I plan ...
0
votes
1answer
301 views
How to use BERT in seq2seq model? [closed]
I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need ...
0
votes
0answers
23 views
How far into the future can I forecast a time-series with an LSTM and strongly seasonal data
I am working on a Sequence-to-Sequence + Attention model for some time-series data. Now I have a really long time series, basically 40 years of daily observations for multiple sensors. The data itself ...
2
votes
1answer
36 views
Build a corpus for machine translation
I want to train an LSTM with attention for translation between French and a "rare" language. I say rare because it is an african language with less digital content, and especially databases ...
1
vote
1answer
29 views
Long range forecasting with sequence-to-sequence models
I have a task where I want to forecast daily observations for 1 year or 2 years in advance at multiple locations--so 365 or 730 days in advance. I actually have a pretty good dataset, meaning daily ...
0
votes
1answer
58 views
Predict customer behaviour with Transformer(attention is all you need)
Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
0
votes
0answers
26 views
How to reduce dimensionality of encoder decoder output?
I have an encoder decoder architecture where the output $ \bar{\bf{y}}_t $ is a sequence of integers of maximum length $n$. Each integer in the sequence is representative of a category so the ...
0
votes
0answers
12 views
Using Transcoder Model for language to language conversion
I have a problem statement like Converting deprecated code into a modern version of the same language. I'm currently converting with a custom Rule-based engine. But the modern version of the language ...
1
vote
1answer
92 views
Why does an attention layer in a transformer learn context?
I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers.
What I'm confused on is why the ...
0
votes
0answers
11 views
Learning conditional statements from natural language
Natural language text in emails might have conditional statements in them. Are there any technical papers and methods that explore converting unstructured text (eg. emails) into structured conditional ...
2
votes
1answer
71 views
How to train a model on top of a transformer to output a sequence?
I am using huggingface to build a model that is capable of identifying mistakes in a given sentence.
Say I have a given sentence and a corresponding label as follows ->
...
2
votes
1answer
25 views
Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)? [closed]
Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors?
If so, say for example we have a sentence and we pass it through the encoder to ...
0
votes
0answers
5 views
Case weights with sequence to sequence models?
I've got a bunch of variable-length sequences. They're basically vectors of zeros with the occasional one. Event rate is maybe 1%.
I want to build a model that will take sequences that I get in the ...
1
vote
1answer
228 views
Pytorch LSTM not training
So I am currently trying to implement an LSTM on Pytorch, but for some reason the loss is not decreasing. Here is my network:
...
0
votes
1answer
76 views
Why does the non autoregresive transfomer model in fairseq require the prev_output_tokens input?
fairseq includes an implementation of a non autoregressive transformer - which (as much as I understand) means that the whole output sequence is generated in a single forward run (in contrast to ...
1
vote
0answers
36 views
When using padding in sequence models, is Keras validation accuracy valid/ reliable?
I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
2
votes
3answers
83 views
What is the best way for synthetic data generation while maintaining privacy?
For one of the projects where we are working as third party contractors, we need a way for the company to share some datasets which can be used for data science. It is not possible for the company to ...
2
votes
0answers
98 views
LSTM low training/validation error but really bad predictions
I'm building a LSTM model to create an automatic drums composer.
I'm following this post: LSTM Metallica
I've built my model and done all the enconding, I was able to emulate the behavior of the ...
1
vote
1answer
131 views
How to add attention mechanism to my sequence-to-sequence architecture in Keras?
Based on this blog entry, I have written a sequence to sequence deep learning model in Keras:
...
1
vote
1answer
43 views
Comparing Language Model of two corpora
I know using Conditional Language Model I can learn the probability of a sentence given the corpus I used to train my model. I will then be able to generate meaningful text by sampling from the ...
0
votes
1answer
283 views
Feeding XLM-R embeddings to neural machine translation?
I’m very new to the field of deep learning. My aim is to make a translation between Catalan to Catalan Sign Language. The grammar of the two languages is different
Input: He sells food.
Output (...
1
vote
0answers
55 views
Multiple time series sequence prediction for multiple multivariate time series
My question is somehow similar to this question, but not satisfied with the answer. I have 100 samples, each sample consists of ...
0
votes
1answer
503 views
Transformer seq2seq model and loading embeddings from XLM-RoBERTa
Is it possible to feed embeddings from XLM- RoBERTa to transformer seq2seq model? I'm working on NMT that translates verbal language sentences to sign language sentences (e.g Input: He sells food. ...