I'm training a Bidirectional LSTM using Keras. My task is to predict the words order in a sentence, so, given a sentence, output of each timestep will be a real number: predicted real numbers of the sentence are ranked in order to obtain integer numbers, indicating the predicted position of the word in the sentence.


sentence -> "Nice I am"

predicted real numbers -> [0.2, 0.6, 0.4]

ranked real numbers -> [3,1,2]

Basically, I pad my sequences to 20, that is the max sequence length found in the dataset.

My model is the following:

model = tf.keras.Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(Bidirectional(LSTM(units=timesteps, return_sequences=True), input_shape=(timesteps, features)))
model.add(Dense(1, activation='linear'))

I mask all the padded vectors to avoid noise in learning. My training loss and my validation loss, during the training, are both low and around 0.33.

In the prediction phase, however, I obtain very bad results. I don't think my model is overfitting/underfitting looking at losses trend.

Is there something bad in my model architecture?

Thanks in advance.

  • $\begingroup$ What do you mean with "I don't think my model is overfitting/underfitting due to losses trend"? Could you elaborate on that? $\endgroup$ – Leevo Apr 17 '20 at 16:35
  • $\begingroup$ @Leevo I know that in general, overfitting is when validation loss is much greater than training loss, and underfitting is when training loss is much greater than validation loss. And looking to my losses, it seems that my model has a good fit. Isn't it? $\endgroup$ – pairon Apr 17 '20 at 16:37
  • $\begingroup$ @Leevo my training set are ordered sentences, but my test set, of course, are unordered sentences. Could shuffle=True improve predictions? $\endgroup$ – pairon Apr 17 '20 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.