Questions tagged [sequence-to-sequence]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
10 views

How to design a writing assistive system that can do writing style guide check like how Grammarly does for gramma checking

I have a style guide that specifies the use of punctuations, line length, and how to break up a line (e.g. before prepositions). I have a dataset that contains original text and the version of the ...
Wayne's user avatar
  • 31
2 votes
1 answer
44 views

How to select the optimal beam size for beam search?

Most Text Generation Models use beam search to select the optimal output candidate. How does one choose the optimal beam size? It would probably vary from task to task, dataset to dataset, and model ...
Tathagato Roy's user avatar
1 vote
0 answers
20 views

How to force my model heads to learn different things?

I have an Seq2Seq model that has 2 generative LM heads. I want the two heads to focus on different features/styles while decoding. The approach that I was thinking of is adding a distance cost to the ...
Tathagato Roy's user avatar
2 votes
0 answers
48 views

Attention mechanisms without a linear layer

I am currently looking into attention mechanism as they are used in (non-Transformer) encoder-decoder architectures, meaning an architecture where some RNN (usually LSTM or GRU) is used in both the ...
krise's user avatar
  • 121
0 votes
0 answers
25 views

Is this problem a time series regression or seq2seq regression or some other type of problem?

I measure sequences of 3 parameters in my system. 2 are independent and the 3rd dependent. Let's call the independent ones $x$ and $y$, and the dependent one $z$. They are each measured once per hour ...
Hitanshu Sachania's user avatar
0 votes
0 answers
587 views

How to Use Generative AI for Time Series Forecasting?

What I have A time series dataset of time stamps (hourly resolution), some covariates (like store foot-traffic) and items sold. What to forecast Number of items sold for next 24 hours, i.e. 24 ...
Della's user avatar
  • 315
0 votes
0 answers
115 views

Seq2seq Transformer Autoencoder returns same results when unfolded using only memory and initial value

I have numeric signals from two sensors, and I would like to create a mapping using sequence-to-sequence autoencoder. I used the transformer architecture, and it seems to be learning - the loss is ...
David Harar's user avatar
0 votes
0 answers
40 views

Which language model to use for this use case? [Finetuning on custom dataset]

An example from my train data is as follows: ...
ksgr5566's user avatar
  • 101
0 votes
0 answers
18 views

Error in seq2seq translation when passing predicted output to rnn due to input shape not always being the same

I'm working on a language translator and I'm getting an error I'm unsure about. During the decoding process when using argmax on the predicted output I am sometimes getting an RuntimeError ...
bailey.bailey's user avatar
0 votes
0 answers
33 views

GRU as a Classifier

Hello Data Scientists, I have sensor data dataset consists of (10 features and labels), the labels classify the data to (normal or 4 type of attacks). The Label is encoded (0,1,2,3,4). There is a ...
Sarah 's user avatar
0 votes
0 answers
29 views

Pretrained model for RNN Encoder-Decoder?

Our team are implementing a paper called Cold-Start-Reinforcement-Learning-with-Softmax-Policy-Gradient. Although the paper didn't mention. We want to use a pre-trained model, which is a RNN Encoder-...
jackson's user avatar
  • 25
1 vote
1 answer
666 views

Why does a decoder generate all hidden states during inference?

Seems that in Vanilla transformers at least (a la AIAYN), during inference time, the hidden states are generated for all tokens in the input sequence, but only the last one is used to predict the next ...
dashnick's user avatar
  • 163
0 votes
0 answers
12 views

In abstractive document summarization task, if I have multiple target sequences for one input document, what is the ideal loss form?

I use autoregressive model like T5 to tackle the abstractive document summarization task. In my case there are multiple target summarizations for one input document. Is there some related works about ...
namespace-Pt's user avatar
0 votes
1 answer
87 views

What is exactly the input to a second lstm layer?

I am often confused about the lstm with more than one layer. Imagine i have two lstm layer with 3 cells each layer. What is exactly the input to the second lstm layer ?
heyoka955's user avatar
0 votes
1 answer
801 views

What are the advantages of autoregressive over seq2seq?

Why are recent dialog agents, such as ChatGPT, BlenderBot3, and Sparrow, based on the decoder architecture instead of the encoder-decoder architecture? I know the difference between the attention of ...
Fei Wang's user avatar
0 votes
0 answers
57 views

I can't figure out why even when training the seq2seq chatbot neural network, it doesn't give adequate answers

When training with 50 thousand pairs of questions and loss 0.2 accuracy 0.9 it does not give adequate answers ...
vamperg's user avatar
0 votes
1 answer
260 views

What loss function to use for predicting discrete output sequence given a discrete input sequence?

I am working on sequence-to-sequence tasks where the input is an n-length sequence of discrete values from a finite set S (say ...
NotNotLogic's user avatar
1 vote
1 answer
1k views

Requirements for variable length output in transformer

I have been working on modifying the transformer from the article The Annotated Transformer. One of the features I would like to include is the ability to pass a sequence of fixed length, and receive ...
try_hard's user avatar
0 votes
1 answer
41 views

Which of these 2 approaches is the best route to learn to build a question answer chatbot?

Quick background on what I am trying to accomplish: I have been working on a project in my company that requires about 300 people across the world to follow quite a large set of rules and guidelines. ...
akkig's user avatar
  • 1
0 votes
0 answers
63 views

Dialogue history encoding for multi-turn dialogues using Seq2seq

In single-turn dialogue seq2seq models where the goal is to produce a good answer y to a query x, sentences are usually encoded such that x is fed to the encoder, while the decoder is only given a &...
postnubilaphoebus's user avatar
0 votes
1 answer
111 views

How to make a pipeline for Videos Dataset for TensorFlow [Sequence Matters] & train Model Effectively with Low Memory System

I am working on a Deep Learning project and I am facing an issue with the size of the dataset. I want to make a pipeline for video dataset [Sequence Matters]. Because if I try the load the whole ...
Sharjeel M. Rajput's user avatar
0 votes
1 answer
58 views

Forecasting of Images using ConvLSTM2D

I am working on a problem of seq2seq modelling using ConvLSTM2D layer in keras. Implementation of convLSTM in keras allows user to control over output sequence using 'return_sequence' option. When ...
user2771151's user avatar
1 vote
0 answers
41 views

RNN to model DNA sequencing classification

I have a DNA sequence dataset each mapped to a certain class. e,g TCAGCCGAGAGCTCATCGATCGTACGT 2 ATGCAGTGCATCGATCGATCGTAGAAC 3 Where the number after the sequence specifies the type of protein this ...
Juliet Soujbel's user avatar
1 vote
1 answer
393 views

Seq2Seq loss function

I was reading the paper neural_approach_conversational_ai.pdf. And in the section Seq2Seq for Text Generation there is a formula that i feel a bit wrong [1]: https://i.stack.imgur.com/sX0it.png Can ...
Văn Hiếu Lê's user avatar
3 votes
1 answer
57 views

String together a set of tokens into a sequence

I have this problem scenario - Given a set of tokens, string them or a subset of the tokens together using stop words into a sequence. I am clear that I can have potentially infinite pre-training data ...
Deepak Saini's user avatar
1 vote
1 answer
529 views

What happens when the length of input is shorter than length of output in transformer architecture?

Given standard transformer architecture with encoder and decoder. What happens when the input for the encoder is shorter than the expected output from the decoder? The decoder is expecting to receive ...
Damian Grzanka's user avatar
0 votes
1 answer
175 views

For an LSTM-based seq2seq model, is reversing the input still necessary or advised when using attention?

The original seq2seq paper reversed the input sequence and cited multiple reasons for doing so. See: Why does LSTM performs better when the source target is reversed? (Seq2seq) But when using ...
Hank's user avatar
  • 3
0 votes
0 answers
158 views

Predict sequence using seqGAN

I am trying to create a GAN model in which I am using this seq2seq as Generator and the following architecture as Discriminator: ...
ksohan's user avatar
  • 53
1 vote
1 answer
407 views

Is it possible to target a specific output length range with BART seq2seq?

I'm currently working on an extractive summary model based on Facebook's BART model. Consistent absolute length output would be highly desirable. The problem is that input length may vary wildly. That ...
Ruben Kruepper's user avatar
1 vote
1 answer
221 views

Multioutput prediction using LSTM encoder decoder with Attention

(I am working on Jupter notebook with python version 3.6.12, running Tensorflow 2.4.0 version.) I have a dataset that consists of 5 input features and 3 output features (that requires to be predicted)....
Sukhmani Kaur Thethi's user avatar
0 votes
1 answer
21 views

Why is sequence prediction always the objective in RNN and LSTM like algorithms

The title is pretty much my question. I haven't seen any literature yet that uses a different training objective. The goal is to find the hidden states eventually, then why is it that only 1 method is ...
huy's user avatar
  • 147
2 votes
0 answers
246 views

Why the LSTM on Keras does not work correctly when it is necessary to predict several steps forward

I used AirPassenger Dataset. And based on several previous values(for examples 20) I want to predict several(3 or 5) steps in future. Like X -> y [10,20,30,....200]->[210,220,230] [20,30,40,.......
Vladimir Shebuniayeu's user avatar
1 vote
1 answer
112 views

In keras seq2seq model, what is the difference between `model.predict()` and the inference model?

I am looking into seq2seq model in keras, for example, this blog post from keras or this. All the examples I have seen have some inference model, that depicts the original model. That inference model ...
BlueMango's user avatar
  • 113
1 vote
0 answers
15 views

Sequence learning from farm operations data

I need to generalize a single sequence from N sequences entailing farming tasks/operations and ultimately plotting it on Gantt chart. There are a total let's assume N sequences = n (total fields) * t (...
yash agarwal's user avatar
1 vote
1 answer
218 views

Multi-step forecasts of factory production data using a Seq2Seq Encoder-Decoder Model with Attention

I am attempting to use a Seq2Seq model to make forecasts of factory production data using an Encoder-Decoder model augmented with Attention. I have become a little stuck as the output of the model ...
InvestingScientist's user avatar
0 votes
1 answer
810 views

Attention weights - change during learning and prediction

Assume a simple LSTM Followed by Attention layer or a full transformer architecture. The attention weights are learnt during training, which get multiplied with keys, queries and values. Please ...
Sandeep Bhutani's user avatar
0 votes
1 answer
41 views

Preprocess multi-sample time series data: encode each sample separately or in aggregate?

Let's say I have 3 dense sequences of uniform length. Should I fit a scaler on them separately or together? ...
Kermit's user avatar
  • 519
1 vote
0 answers
41 views

Workaround / fallback value for tfp.distributions.Categorical.log_prob in tensorflow graph mode

Is there a way to avoid tfp.distributions.Categorical.log_probraising an error if the input is a label out of range? I am passing a batch of samples to the ...
RR_28023's user avatar
1 vote
1 answer
1k views

Calculating confidence score in NER

I am working on a problem on Named Entity Recognition. Given a text, my model is detecting the Named Entities and extracting that info for the end-user. Now the ask is end-user needs a confidence ...
Saikat Bhattacharya's user avatar
3 votes
1 answer
1k views

Multi-output, multi-timestep sequence prediction with Keras

I've been searching for about three hours and I can't find an answer to a very simple question. I have a time series prediction problem. I am trying to use a Keras LSTM model (with a Dense at the end) ...
MGreen's user avatar
  • 31
3 votes
1 answer
1k views

A simple attention based text prediction model from scratch using pytorch

I first asked this question in codereview SE but a user recommended to post this here instead. I have created a simple self attention based text prediction model using pytorch. The attention formula ...
Eka's user avatar
  • 301
1 vote
1 answer
41 views

How to implement sequence to sequence models?

I have a dataset with patient demographics, diagnosis history, hospital visit dates, drugs consumed etc. All these events have time stamp information (except static info like demographics such gender, ...
The Great's user avatar
  • 2,565
3 votes
1 answer
3k views

Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder

According to https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html, the forward function of nn.TransformerDecoder contemplates the following arguments: tgt – the sequence to the ...
Pablo Messina's user avatar
0 votes
1 answer
126 views

Different training method for encoder-decoder model

Trying to learn the encoder-decoder model for some NLP problems. I am referring to this Keras tutorial. During the model training phase, this tutorial just uses the following: ...
data_person's user avatar
7 votes
1 answer
9k views

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.). For the purposes of this question, you may assume that I already have the input part ...
Pablo Messina's user avatar
1 vote
0 answers
83 views

Dummy Variables of Weights in RNN Backpropagation Through Time

In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that - To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are defined to ...
KKGanguly's user avatar
  • 111
4 votes
1 answer
2k views

How does the Transformer predict n steps into the future?

I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...
skevelis's user avatar
0 votes
2 answers
578 views

Difference between zero-padding and character-padding in Recurrent Neural Networks

For RNN's to work efficiently we vectorize the problem which results in an input matrix of shape (m, max_seq_len) where m is the number of examples, e.g. ...
PhysicsMan's user avatar
2 votes
2 answers
442 views

Timeseries LSTM: does test data need to come after training data?

I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence). Because the time series is very long, I plan ...
lodo's user avatar
  • 121
1 vote
1 answer
2k views

How to use BERT in seq2seq model? [closed]

I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need ...
Andrey's user avatar
  • 113