Questions tagged [sequence-to-sequence]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

RNN to model DNA sequencing classification

I have a DNA sequence dataset each mapped to a certain class. e,g TCAGCCGAGAGCTCATCGATCGTACGT 2 ATGCAGTGCATCGATCGATCGTAGAAC 3 Where the number after the sequence specifies the type of protein this ...
user avatar
0 votes
0 answers
6 views

Atomic tasks from a complex task using NLP

I have a problem statement when I need to find all the tasks that the server had to do based on a complex task. Example, in a 3D modeling scenario, if the model is queried with a complex task such as &...
user avatar
  • 41
1 vote
1 answer
23 views

Seq2Seq loss function

I was reading the paper neural_approach_conversational_ai.pdf. And in the section Seq2Seq for Text Generation there is a formula that i feel a bit wrong [1]: https://i.stack.imgur.com/sX0it.png Can ...
user avatar
3 votes
1 answer
38 views

String together a set of tokens into a sequence

I have this problem scenario - Given a set of tokens, string them or a subset of the tokens together using stop words into a sequence. I am clear that I can have potentially infinite pre-training data ...
user avatar
0 votes
0 answers
55 views

How to implement Early stopping in Neural Machine Translation with Attention or Transformers?

I am trying to implement early stopping to my model where I am performing Machine Translation using Seq2Seq with attention. I am mostly used to writing my own models in steps, something like this: <...
user avatar
0 votes
1 answer
52 views

What happens when the length of input is shorter than length of output in transformer architecture?

Given standard transformer architecture with encoder and decoder. What happens when the input for the encoder is shorter than the expected output from the decoder? The decoder is expecting to receive ...
user avatar
0 votes
0 answers
9 views

Multivariate Multistep Prediction for Variable Length of Input and Output Sequence

I want to develop recurrent models (RNN, LSTM, etc.) to predict the output variable for the rest of timesteps. I have 10,000 samples/users. Each user has multivariate sequences (50 features). So the ...
user avatar
  • 1
0 votes
1 answer
29 views

For an LSTM-based seq2seq model, is reversing the input still necessary or advised when using attention?

The original seq2seq paper reversed the input sequence and cited multiple reasons for doing so. See: Why does LSTM performs better when the source target is reversed? (Seq2seq) But when using ...
user avatar
  • 3
0 votes
0 answers
57 views

How Transformer Decoders Use Mask to Prevent Ground Truth Leakage During Training process

After reading the original paper and many articles and blogs, I have a general understanding of Transformer. I still have some doubts about the Mask, I know it is to prevent the subsequent positions ...
user avatar
  • 1
0 votes
0 answers
54 views

Predict sequence using seqGAN

I am trying to create a GAN model in which I am using this seq2seq as Generator and the following architecture as Discriminator: ...
user avatar
  • 53
0 votes
0 answers
62 views

LSTM behaviour with return_sequences and TimeDistributed

I am trying different models for a classification problem with sequence data and variable sequence length, the below model predict all at once, and it achieve better results than other models, so I ...
user avatar
  • 1
0 votes
0 answers
32 views

Using sequences for multilabel classification

I have a sequential dataset of events, which looks like the following: ['some text here', 'more text here'] -> target Each datapoint is a true sequence ...
user avatar
  • 161
0 votes
1 answer
86 views

Is it possible to target a specific output length range with BART seq2seq?

I'm currently working on an extractive summary model based on Facebook's BART model. Consistent absolute length output would be highly desirable. The problem is that input length may vary wildly. That ...
user avatar
0 votes
0 answers
17 views

Passing both x_test and y_test to model.predict() in a sequence-to-sequence model?

I have a sequence to sequence model for text summarization like this: model = Model([encoder_inputs, decoder_inputs], decoder_dense) I fit it accordingly: ...
user avatar
1 vote
1 answer
56 views

Multioutput prediction using LSTM encoder decoder with Attention

(I am working on Jupter notebook with python version 3.6.12, running Tensorflow 2.4.0 version.) I have a dataset that consists of 5 input features and 3 output features (that requires to be predicted)....
user avatar
0 votes
0 answers
11 views

Composite Input into Seq2Seq LSTM Network

Given that we have a seq2seq problem, where the input sequence is indeed multiple inputs and not only one as in traditional seq2seq problems. For example, in language translation, we usually give ...
user avatar
  • 179
0 votes
0 answers
26 views

What is the SOTA in Pointer Networks?

I have more or less a sequence2sequence generation task. The special nature is that the correct answer is guaranteed to be a combination of the tokens in the input. ...
user avatar
0 votes
0 answers
35 views

Autoencoder in keras and accuracy

I am looking at Autoencoders in keras. They say, "Autoencoding" is a data compression algorithm where the compression and decompression functions are 1) data-specific, 2) lossy, and 3) ...
user avatar
  • 103
0 votes
1 answer
15 views

Why is sequence prediction always the objective in RNN and LSTM like algorithms

The title is pretty much my question. I haven't seen any literature yet that uses a different training objective. The goal is to find the hidden states eventually, then why is it that only 1 method is ...
user avatar
  • 37
2 votes
0 answers
62 views

Why the LSTM on Keras does not work correctly when it is necessary to predict several steps forward

I used AirPassenger Dataset. And based on several previous values(for examples 20) I want to predict several(3 or 5) steps in future. Like X -> y [10,20,30,....200]->[210,220,230] [20,30,40,.......
user avatar
0 votes
0 answers
12 views

Sequence to Sequence learning applied to list of numbers

I am looking to apply ML methods to genetic data. My goal is to predict which rare (generally de novo) mutations a person has based on what non-rare (generally inherited) mutations. I have worked on ...
user avatar
  • 101
0 votes
0 answers
54 views

Empty prediction with keras Seq2Seq with attention mechanism

I have a simply seq2seq model with attention mechanism in keras. My problem is that the inference model only gives me empty prediction. However, if I remove the attention it suvessfully gives me the ...
user avatar
  • 103
0 votes
1 answer
27 views

In keras seq2seq model, what is the difference between `model.predict()` and the inference model?

I am looking into seq2seq model in keras, for example, this blog post from keras or this. All the examples I have seen have some inference model, that depicts the original model. That inference model ...
user avatar
  • 103
1 vote
0 answers
10 views

Sequence learning from farm operations data

I need to generalize a single sequence from N sequences entailing farming tasks/operations and ultimately plotting it on Gantt chart. There are a total let's assume N sequences = n (total fields) * t (...
user avatar
1 vote
1 answer
126 views

Multi-step forecasts of factory production data using a Seq2Seq Encoder-Decoder Model with Attention

I am attempting to use a Seq2Seq model to make forecasts of factory production data using an Encoder-Decoder model augmented with Attention. I have become a little stuck as the output of the model ...
user avatar
0 votes
1 answer
137 views

Attention weights - change during learning and prediction

Assume a simple LSTM Followed by Attention layer or a full transformer architecture. The attention weights are learnt during training, which get multiplied with keys, queries and values. Please ...
user avatar
0 votes
1 answer
32 views

Preprocess multi-sample time series data: encode each sample separately or in aggregate?

Let's say I have 3 dense sequences of uniform length. Should I fit a scaler on them separately or together? ...
user avatar
  • 476
1 vote
0 answers
16 views

Workaround / fallback value for tfp.distributions.Categorical.log_prob in tensorflow graph mode

Is there a way to avoid tfp.distributions.Categorical.log_probraising an error if the input is a label out of range? I am passing a batch of samples to the ...
user avatar
0 votes
1 answer
769 views

Calculating confidence score in NER

I am working on a problem on Named Entity Recognition. Given a text, my model is detecting the Named Entities and extracting that info for the end-user. Now the ask is end-user needs a confidence ...
user avatar
3 votes
1 answer
444 views

Multi-output, multi-timestep sequence prediction with Keras

I've been searching for about three hours and I can't find an answer to a very simple question. I have a time series prediction problem. I am trying to use a Keras LSTM model (with a Dense at the end) ...
user avatar
  • 31
2 votes
1 answer
378 views

A simple attention based text prediction model from scratch using pytorch

I first asked this question in codereview SE but a user recommended to post this here instead. I have created a simple self attention based text prediction model using pytorch. The attention formula ...
user avatar
  • 271
1 vote
1 answer
31 views

How to implement sequence to sequence models?

I have a dataset with patient demographics, diagnosis history, hospital visit dates, drugs consumed etc. All these events have time stamp information (except static info like demographics such gender, ...
user avatar
  • 2,215
1 vote
1 answer
611 views

Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder

According to https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html, the forward function of nn.TransformerDecoder contemplates the following arguments: tgt – the sequence to the ...
user avatar
0 votes
1 answer
22 views

Different training method for encoder-decoder model

Trying to learn the encoder-decoder model for some NLP problems. I am referring to this Keras tutorial. During the model training phase, this tutorial just uses the following: ...
user avatar
5 votes
1 answer
2k views

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.). For the purposes of this question, you may assume that I already have the input part ...
user avatar
1 vote
0 answers
46 views

Dummy Variables of Weights in RNN Backpropagation Through Time

In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that - To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are defined to ...
user avatar
  • 111
2 votes
1 answer
346 views

How does the Transformer predict n steps into the future?

I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...
user avatar
0 votes
2 answers
155 views

Difference between zero-padding and character-padding in Recurrent Neural Networks

For RNN's to work efficiently we vectorize the problem which results in an input matrix of shape (m, max_seq_len) where m is the number of examples, e.g. ...
user avatar
2 votes
1 answer
190 views

Timeseries LSTM: does test data need to come after training data?

I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence). Because the time series is very long, I plan ...
user avatar
  • 121
1 vote
1 answer
1k views

How to use BERT in seq2seq model? [closed]

I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need ...
user avatar
  • 113
2 votes
1 answer
102 views

Build a corpus for machine translation

I want to train an LSTM with attention for translation between French and a "rare" language. I say rare because it is an african language with less digital content, and especially databases ...
user avatar
1 vote
1 answer
38 views

Long range forecasting with sequence-to-sequence models

I have a task where I want to forecast daily observations for 1 year or 2 years in advance at multiple locations--so 365 or 730 days in advance. I actually have a pretty good dataset, meaning daily ...
user avatar
  • 153
0 votes
1 answer
106 views

Predict customer behaviour with Transformer(attention is all you need)

Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
user avatar
0 votes
0 answers
27 views

How to reduce dimensionality of encoder decoder output?

I have an encoder decoder architecture where the output $ \bar{\bf{y}}_t $ is a sequence of integers of maximum length $n$. Each integer in the sequence is representative of a category so the ...
user avatar
  • 101
1 vote
1 answer
167 views

Why does an attention layer in a transformer learn context?

I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers. What I'm confused on is why the ...
user avatar
2 votes
1 answer
134 views

How to train a model on top of a transformer to output a sequence?

I am using huggingface to build a model that is capable of identifying mistakes in a given sentence. Say I have a given sentence and a corresponding label as follows -> ...
user avatar
2 votes
1 answer
28 views

Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)? [closed]

Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors? If so, say for example we have a sentence and we pass it through the encoder to ...
user avatar
  • 23
1 vote
1 answer
508 views

Pytorch LSTM not training

So I am currently trying to implement an LSTM on Pytorch, but for some reason the loss is not decreasing. Here is my network: ...
user avatar
0 votes
1 answer
130 views

Why does the non autoregresive transfomer model in fairseq require the prev_output_tokens input?

fairseq includes an implementation of a non autoregressive transformer - which (as much as I understand) means that the whole output sequence is generated in a single forward run (in contrast to ...
user avatar
3 votes
1 answer
69 views

When using padding in sequence models, is Keras validation accuracy valid/ reliable?

I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
user avatar