Questions tagged [sequence-to-sequence]
The sequence-to-sequence tag has no usage guidance.
117
questions
0
votes
1
answer
22
views
What is exactly the input to a second lstm layer?
I am often confused about the lstm with more than one layer.
Imagine i have two lstm layer with 3 cells each layer.
What is exactly the input to the second lstm layer ?
0
votes
1
answer
48
views
What are the advantages of autoregressive over seq2seq?
Why are recent dialog agents, such as ChatGPT, BlenderBot3, and Sparrow, based on the decoder architecture instead of the encoder-decoder architecture?
I know the difference between the attention of ...
- 3
0
votes
0
answers
35
views
I can't figure out why even when training the seq2seq chatbot neural network, it doesn't give adequate answers
When training with 50 thousand pairs of questions and loss 0.2 accuracy 0.9 it does not give adequate answers
...
- 1
0
votes
1
answer
31
views
What loss function to use for predicting discrete output sequence given a discrete input sequence?
I am working on sequence-to-sequence tasks where the input is an n-length sequence of discrete values from a finite set S (say ...
0
votes
0
answers
15
views
Batch inference and seq2seq
I am following this guide to the Seq2Seq architecture. It is clear that the author suggests batch training during the teacher forcing step. However, during the inference step, the author restricts the ...
- 123
0
votes
1
answer
49
views
Requirements for variable length output in transformer
I have been working on modifying the transformer from the article The Annotated Transformer. One of the features I would like to include is the ability to pass a sequence of fixed length, and receive ...
- 1
0
votes
1
answer
21
views
Which of these 2 approaches is the best route to learn to build a question answer chatbot?
Quick background on what I am trying to accomplish:
I have been working on a project in my company that requires about 300 people across the world to follow quite a large set of rules and guidelines. ...
- 1
0
votes
0
answers
30
views
Siamese network for a sequence-to-sequence generation
Shall I use the siamese network for a sequence-to-sequence generation problem in machine learning?
Eg:
Input 1: Sentence 1 (sequence)
Input 2: Sentence 2 (sequence)
Output: Newly Generated sentence (...
- 1
0
votes
0
answers
32
views
Dialogue history encoding for multi-turn dialogues using Seq2seq
In single-turn dialogue seq2seq models where the goal is to produce a good answer y to a query x, sentences are usually encoded such that x is fed to the encoder, while the decoder is only given a &...
0
votes
1
answer
47
views
How to make a pipeline for Videos Dataset for TensorFlow [Sequence Matters] & train Model Effectively with Low Memory System
I am working on a Deep Learning project and I am facing an issue with the size of the dataset. I want to make a pipeline for video dataset [Sequence Matters]. Because if I try the load the whole ...
- 21
0
votes
0
answers
144
views
SOS / EOS tokens to encode fixed-size 1D real and complex-valued short signals with seq2seq
Are there any standard or recommended SOS (start of sequence) and EOS (end of sequence) tokens for seq2seq encoding using RNN/LSTM/Transformers applied to real-valued and/or complex-valued 1D signals (...
- 101
0
votes
0
answers
17
views
Fixing/up-rezzing images is easily possible these days in practice. Is there any analogous tool available for low-quality audio?
Tools like waifu2x and Anime4k (for the MPV video player) can already easily de-noise and up-rez low quality images to 4k quality pretty darn well. Similar tools like DLSS and FSR are used for video ...
- 111
0
votes
1
answer
27
views
Forecasting of Images using ConvLSTM2D
I am working on a problem of seq2seq modelling using ConvLSTM2D layer in keras. Implementation of convLSTM in keras allows user to control over output sequence using 'return_sequence' option. When ...
- 101
0
votes
0
answers
36
views
What does logits in Casual Language Modeling represent?
I am reading the docs for transformers by hugging face and I see that the logits produced by casual language models are of the shape (batch_size, sequence_length, config.vocab_size).
I also read the ...
0
votes
0
answers
31
views
Image captioning using sequence to sequence model
I am trying to build a sequence to sequence model. I have used Vision transformer as the encoder and LSTM with 1 layers as the decoder. The output of the encoder is given as the hidden state for ...
0
votes
0
answers
16
views
Given an audio file, output a processed audio file (with something like seq2seq)
I have a dataset of audio files of varying durations. Given audio file x, I would like to get audio file y (audio files ...
- 1
0
votes
0
answers
44
views
How to use batches for training in Sequence-to-Sequence models?
I am working on a seq-to-seq Image Captioning model, with Vision Transformer as the encoder and a LSTM based model as a decoder. The output from the encoder is given as the hidden state and cell state ...
1
vote
0
answers
21
views
RNN to model DNA sequencing classification
I have a DNA sequence dataset each mapped to a certain class.
e,g
TCAGCCGAGAGCTCATCGATCGTACGT 2
ATGCAGTGCATCGATCGATCGTAGAAC 3
Where the number after the sequence specifies the type of protein this ...
0
votes
0
answers
8
views
Atomic tasks from a complex task using NLP
I have a problem statement when I need to find all the tasks that the server had to do based on a complex task. Example, in a 3D modeling scenario, if the model is queried with a complex task such as &...
- 41
1
vote
1
answer
98
views
Seq2Seq loss function
I was reading the paper neural_approach_conversational_ai.pdf. And in the section Seq2Seq for Text Generation there is a formula that i feel a bit wrong
[1]: https://i.stack.imgur.com/sX0it.png
Can ...
- 11
3
votes
1
answer
50
views
String together a set of tokens into a sequence
I have this problem scenario - Given a set of tokens, string them or a subset of the tokens together using stop words into a sequence. I am clear that I can have potentially infinite pre-training data ...
- 141
0
votes
1
answer
260
views
What happens when the length of input is shorter than length of output in transformer architecture?
Given standard transformer architecture with encoder and decoder.
What happens when the input for the encoder is shorter than the expected output from the decoder?
The decoder is expecting to receive ...
- 101
0
votes
1
answer
83
views
For an LSTM-based seq2seq model, is reversing the input still necessary or advised when using attention?
The original seq2seq paper reversed the input sequence and cited multiple reasons for doing so. See:
Why does LSTM performs better when the source target is reversed? (Seq2seq)
But when using ...
- 3
0
votes
0
answers
108
views
Predict sequence using seqGAN
I am trying to create a GAN model in which I am using this seq2seq as Generator and the following architecture as Discriminator:
...
- 53
1
vote
1
answer
266
views
Is it possible to target a specific output length range with BART seq2seq?
I'm currently working on an extractive summary model based on Facebook's BART model. Consistent absolute length output would be highly desirable. The problem is that input length may vary wildly. That ...
- 121
1
vote
1
answer
139
views
Multioutput prediction using LSTM encoder decoder with Attention
(I am working on Jupter notebook with python version 3.6.12, running Tensorflow 2.4.0 version.)
I have a dataset that consists of 5 input features and 3 output features (that requires to be predicted)....
0
votes
1
answer
16
views
Why is sequence prediction always the objective in RNN and LSTM like algorithms
The title is pretty much my question. I haven't seen any literature yet that uses a different training objective. The goal is to find the hidden states eventually, then why is it that only 1 method is ...
- 37
2
votes
0
answers
129
views
Why the LSTM on Keras does not work correctly when it is necessary to predict several steps forward
I used AirPassenger Dataset. And based on several previous values(for examples 20) I want to predict several(3 or 5) steps in future.
Like
X -> y
[10,20,30,....200]->[210,220,230]
[20,30,40,.......
1
vote
1
answer
68
views
In keras seq2seq model, what is the difference between `model.predict()` and the inference model?
I am looking into seq2seq model in keras, for example, this blog post from keras or this. All the examples I have seen have some inference model, that depicts the original model. That inference model ...
- 113
1
vote
0
answers
15
views
Sequence learning from farm operations data
I need to generalize a single sequence from N sequences entailing farming tasks/operations and ultimately plotting it on Gantt chart.
There are a total let's assume N sequences = n (total fields) * t (...
- 11
1
vote
1
answer
172
views
Multi-step forecasts of factory production data using a Seq2Seq Encoder-Decoder Model with Attention
I am attempting to use a Seq2Seq model to make forecasts of factory production data using an Encoder-Decoder model augmented with Attention. I have become a little stuck as the output of the model ...
0
votes
1
answer
428
views
Attention weights - change during learning and prediction
Assume a simple LSTM Followed by Attention layer or a full transformer architecture. The attention weights are learnt during training, which get multiplied with keys, queries and values.
Please ...
- 864
0
votes
1
answer
36
views
Preprocess multi-sample time series data: encode each sample separately or in aggregate?
Let's say I have 3 dense sequences of uniform length. Should I fit a scaler on them separately or together?
...
- 479
1
vote
0
answers
30
views
Workaround / fallback value for tfp.distributions.Categorical.log_prob in tensorflow graph mode
Is there a way to avoid tfp.distributions.Categorical.log_probraising an error if the input is a label out of range?
I am passing a batch of samples to the ...
- 11
1
vote
1
answer
1k
views
Calculating confidence score in NER
I am working on a problem on Named Entity Recognition. Given a text, my model is detecting the Named Entities and extracting that info for the end-user. Now the ask is end-user needs a confidence ...
3
votes
1
answer
832
views
Multi-output, multi-timestep sequence prediction with Keras
I've been searching for about three hours and I can't find an answer to a very simple question.
I have a time series prediction problem. I am trying to use a Keras LSTM model (with a Dense at the end) ...
- 31
2
votes
1
answer
766
views
A simple attention based text prediction model from scratch using pytorch
I first asked this question in codereview SE but a user recommended to post this here instead.
I have created a simple self attention based text prediction model using pytorch. The attention formula ...
- 281
1
vote
1
answer
33
views
How to implement sequence to sequence models?
I have a dataset with patient demographics, diagnosis history, hospital visit dates, drugs consumed etc.
All these events have time stamp information (except static info like demographics such gender, ...
- 2,449
3
votes
1
answer
1k
views
Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder
According to https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html, the forward function of nn.TransformerDecoder contemplates the following arguments:
tgt – the sequence to the ...
- 347
0
votes
1
answer
38
views
Different training method for encoder-decoder model
Trying to learn the encoder-decoder model for some NLP problems.
I am referring to this Keras tutorial.
During the model training phase, this tutorial just uses the following:
...
- 215
5
votes
1
answer
4k
views
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.).
For the purposes of this question, you may assume that I already have the input part ...
- 347
1
vote
0
answers
73
views
Dummy Variables of Weights in RNN Backpropagation Through Time
In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that -
To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are defined to ...
- 111
2
votes
1
answer
976
views
How does the Transformer predict n steps into the future?
I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...
- 23
0
votes
2
answers
328
views
Difference between zero-padding and character-padding in Recurrent Neural Networks
For RNN's to work efficiently we vectorize the problem which results in an input matrix of shape
(m, max_seq_len)
where m is the number of examples, e.g. ...
2
votes
1
answer
273
views
Timeseries LSTM: does test data need to come after training data?
I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence).
Because the time series is very long, I plan ...
- 121
1
vote
1
answer
2k
views
How to use BERT in seq2seq model? [closed]
I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need ...
- 113
2
votes
1
answer
171
views
Build a corpus for machine translation
I want to train an LSTM with attention for translation between French and a "rare" language. I say rare because it is an african language with less digital content, and especially databases ...
- 23
1
vote
1
answer
42
views
Long range forecasting with sequence-to-sequence models
I have a task where I want to forecast daily observations for 1 year or 2 years in advance at multiple locations--so 365 or 730 days in advance. I actually have a pretty good dataset, meaning daily ...
- 153
0
votes
1
answer
154
views
Predict customer behaviour with Transformer(attention is all you need)
Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
0
votes
0
answers
27
views
How to reduce dimensionality of encoder decoder output?
I have an encoder decoder architecture where the output $ \bar{\bf{y}}_t $ is a sequence of integers of maximum length $n$. Each integer in the sequence is representative of a category so the ...
- 101