Questions tagged [sequence-to-sequence]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
19 views

Feeding XLM-R embeddings to neural machine translation?

I’m very new to the field of deep learning. My aim is to make a translation between Catalan to Catalan Sign Language. The grammar of the two languages is different Input: He sells food. Output (...
0
votes
0answers
11 views

Multiple time series sequence prediction for multiple multivariate time series

My question is somehow similar to this question, but not satisfied with the answer. I have 100 samples, each sample consists of ...
0
votes
1answer
21 views

Transformer seq2seq model and loading embeddings from XLM-RoBERTa

Is it possible to feed embeddings from XLM- RoBERTa to transformer seq2seq model? I'm working on NMT that translates verbal language sentences to sign language sentences (e.g Input: He sells food. ...
1
vote
0answers
14 views

Seq2Seq for sentence correction

I have a task in hand where I get a dirty formed sentence and need to correct it. Examples are, "StackOverflow best question answering platform" to be converted to "StackOverflow is best question ...
1
vote
1answer
12 views

Sequence labeling with partially known labels

I am working on a sequence labeling task where, based on experience, many of the labels of a given input sequence can be reliably extracted with a simple rule-based approach. For example, considering ...
0
votes
0answers
18 views

Seeking your advice on XLM-R for NMT

I want to use XLM-R for neural machine translation b/n the same low resource language? For example: Input-> He sells food(in Catalan) Output-> Food he sells(in Catalan) Anyone having code example/...
1
vote
2answers
22 views

Which input to use when generating a new sequence

I want to use sequence-to-sequence architecture to generate sequences. My data has such structure ...
1
vote
0answers
47 views

How can I finetune XLM-R for neural machine translation between the same language(Catalan to Catalan-with different grammar structure))?

Can I fine-tune XLM-R to do something similar to https://www.guru99.com/seq2seq-model.html (encoder-decoder architecture or any other way that perform good in small dataset - around 1000) For example: ...
1
vote
1answer
142 views

Can I fine-tune BERT, ELMO or XLnet for Seq2Seq neural machine translation?

I'm working on neural machine translator that translates English sentences to American sign language sentences(e.g below). I've a quite small dataset - around 1000 sentence pairs. I'm wondering if it ...
0
votes
1answer
143 views

Is it possible feed BERT to seq2seq encoder/decoder NMT (for low resource language)?

I'm working on NMT model which the input and the target sentences are from the same language (but the grammar differs). I'm planning to pre-train and use BERT since I'm working on small dataset and ...
0
votes
2answers
37 views

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
2
votes
1answer
34 views

Methods for learning with noisy labels

I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels. More precisely, I would like this method to be able to leverage noisy data as ...
1
vote
0answers
10 views

How can I do a sequence to sequence model (RNN / LSTM) with Keras with fixed length data?

What I'm trying to do seems so simple, but I can't find any examples online. First, I'm not working in language, so all of the embedding stuff adds needless ...
3
votes
2answers
65 views

Sentences language translation with neural network, with a simple layer structure (if possible sequential)

Context: Many language sentences translation systems (e.g. French to English) with neural networks use a seq2seq structure: "the cat sat on the mat" -> [Seq2Seq ...
1
vote
1answer
31 views

Answer to Question

Looking for a system which can generate answers to questions. Most systems and blogs posted on internet are on Question to answer but not on answer to question or paraphrasing or keyword to questions. ...
1
vote
0answers
550 views

SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors

I am writing Encoder-Decoder architecture with Bahdanau Attention using tf.keras with TensorFlow 2.0. Below is my code This is working with TensorFlow 1.15 but getting the error in 2.0. you can check ...
1
vote
1answer
606 views

How can I feed BERT to neural machine translation?

I am trying to feed the input and target sentences to an NMT model, I am trying to use BERT here, But I don't have any idea how ...
1
vote
0answers
44 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
1
vote
1answer
233 views

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
0
votes
0answers
8 views

Output of a seq2seq network

I've created a seq2seq lstm that uses word embeddings, following the tutorial in tutorial , but I'm having some problems understanding the output, which is along these lines: ...
1
vote
0answers
27 views

Improving Performance of LSTM for time series prediction

My data consists of two features and a set of time series data labeled as "bookings". I have 1056 data point in the times series, for which I have two features for each. The data size is 1056x3. My ...
0
votes
0answers
148 views

Adapting Pytorch tutorial “NMT from Scratch…” for dynamic RNN

I have taken the code from the tutorial and attempted to modify it to include bi-directionality and any arbitrary numbers of layers for GRU. Link to the tutorial which uses uni-directional, single ...
0
votes
0answers
22 views

two different attention methods for seq2seq

I see two different ways of applying attention in seq2seq: (a) the context vector (the weighted sum of encoder hidden states) fed into the output softmax, as shown in the diagram below. The diagram ...
0
votes
0answers
44 views

Neural Network for text generation

I'm new into ML and NN and I would like to figure out what is the best way to solve this (initially little) problem. Suppose i have a {number} in input and i want as output the phrase "The number in ...
0
votes
0answers
36 views

What's the meaning of having a UNK token for out of vocabulary words during decoding?

Adding a UNK token to the vocabulary is a conventional way to handle oov works in tasks of NLP. It is totally understandable to have it for encoding, but what's the ...
1
vote
0answers
14 views

How to create a seq2seq without specifying a fixed decoder length?

Based on the model presented in this answer: ...
0
votes
0answers
67 views

Scheduled Sampling in Keras

Im wondering how scheduled sampling (maybe together with curriculum learning) as described in this paper [https://arxiv.org/abs/1506.03099] could be implemented in keras. Lets assume a simple ...
0
votes
1answer
35 views

How can I build a seq2seq model , which is topic aware

I have developed a chatbot, which is basically a seq2seq LSTM network. Which can generate text based on input text. But the problem I am having right now is it is not topic aware. As an example : ...
0
votes
1answer
7 views

Confusion about Decoder labels for training seq-to-seq models

So in seq-to-seq models for say NMT, the decoder is a sequence model for the right-shifted intended output. My question is, during training, are the inputs and outputs of the decoder supposed to be ...
1
vote
0answers
154 views

Why isnt my seq2seq model reconising what the <END> tag is?

I am making a seq2seq chatbot with the help of this guide: https://medium.com/predict/creating-a-chatbot-from-scratch-using-keras-and-tensorflow-59e8fc76be79 . I set up the model and data processing ...
0
votes
0answers
17 views

Encoding numbers and words

I am fairly new to seq2seq models in nlp and just really learned about them. Anyway, in many of the examples, I have seen there has been to approaches to providing a model with data. One in which is ...
0
votes
0answers
119 views

Test case generation from user stories using NLP and NLG

I have an existing database with User stories (with multiple text fields like Description with Use cases, acceptance criteria etc) and corresponding test cases with steps for each test case. I am ...
0
votes
0answers
26 views

Which algorithm to use for path to prescription?

I am working on a business problem in the commercial pharma industry. In the pharma industry, we have medical representatives (Reps) selling drugs to health care providers (HCPs). They frequently ...
0
votes
0answers
90 views

Error propagation in Time series forecast with many-to-many multi-steps RNN/LSTM

I am trying to do a many-to-many time series forecast, which features an encoder-decoder model to predict with variable input and fixed prediction period. In my case, I want to predict for the future ...
0
votes
0answers
21 views

Deep Learning - Predict the relative order of data

I am facing a problem in which i want to predict the order of data. I was searching for research papers, however i do not know how this problem is named in academia. I encountered the following well ...
0
votes
1answer
44 views

Seq2Seq Model training: Encoder vs. Decoder

Can someone point me to an article which explains how the model training is done in Seq2Seq? I know "Teacher Forcing" is used but what I found so far hasn't been detailed enough. What I am most ...
1
vote
1answer
2k views

IndexError: list index out of range

I'm implementing a sequence-2-sequence model with RNN-VAE architecture, and I use an attention mechanism. I have problem in the decoder part. I'm struggling with this error: IndexError: list index ...
0
votes
0answers
303 views

ValueError: Dimensions must be equal, but are 256 and 12 for 'attention_layer/MatMul_1' (op: 'MatMul') with input shapes: [?,256], [12,256]

I'm working on a sequence-to-sequence approach using LSTM and a VAE with an attention mechanism. ...
2
votes
0answers
670 views

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism. ...
0
votes
0answers
23 views

Strategy for “forcing” number of labels in seq2seq predictions with Keras?

I'm trying to train a seq2seq model that for every timestep in a given timeseries sample will output 1 of 6 possible labels. Furthermore, the training data is constructed in such a way that Each ...
0
votes
0answers
34 views

What encoding to use for my musical vectors?

I'm trying to build a music recommendations system using an encoder-decoder sequence-to-sequence architecture using keras. My dataset comprises of playlists containing songs represented as a 13-...
2
votes
1answer
62 views

Working ofLSTM with multiple Units - NER

I am trying to understand working of LSTM networks and kind of not clear about how different neurons in a cell interact each other. I had a look at a similar question, but still not clear about few ...
1
vote
0answers
26 views

Adding context in a sequence to sequence problem

The encoder of a seq2seq model is meant to generate a conditioning context for the decoder, as mentioned here A RNN layer (or stack thereof) acts as "encoder": it processes the input sequence and ...
3
votes
1answer
2k views

One-hot encode multi-class multi-label sequences

Suppose I want to build a timeseries where each timestep is represented by a categorical array: the encoded sequences look like [[2, 0, 5],[3, 1, 4],..] and each ...
1
vote
0answers
30 views

Initialising states in a multilayer sequence to sequence model

With a sequence to sequence model where the enocoder and decoder are both comprised of one layer each, the initial state of the decoder is initialised to use the final states of the encoder layer. In ...
1
vote
0answers
110 views

What should the size of the decoder output be in a sequence to sequence model

In a sequence to sequence model, a lot of the tutorials I have read state that the decoder target length should be the same as the encoder input length (https://blog.keras.io/building-autoencoders-in-...
2
votes
1answer
8k views

How to determine feature importance in a neural network?

I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the ...
2
votes
0answers
581 views

Recommended model for univariate or multivariate multistep ahead time series forecasting

I have a dataset consisting of recurring and non-recurring expense transactions from bank accounts, as well as other features describing the bank account and each transation. I aggregate these ...
1
vote
1answer
2k views

Predicting next number in a sequence - data analysis

I am a machine learning newbie and I am working on a project where I'm given a sequence of integers all of which are in the range 0 to 70. My goal is to predict the next integer in the sequence given ...
2
votes
2answers
4k views

Does this encoder-decoder LSTM make sense for time series sequence to sequence?

TASK given $\vec x = [x_{t=-3}, x_{t=-2}, x_{t=-1}, x_{t=0}]$ predict $\vec y = [x_{t=1}, x_{t=2}]$ Whith an LSTM encoder-decoder (seq2seq) MODEL NOTE: the ? symbol in the shape of the tensors ...