0
$\begingroup$

I am attempting to construct a Keras model that intakes a sequence of vectors and outputs the most likely next vector in the sequence. I have followed a few tutorials, but nothing is quite seeming to work. My accuracy is always extremely low, sometimes even 0, and loss increases.

My model is as follows.

model = keras.Sequential()
model.add(layers.LSTM(300, return_sequences=True))  
model.add(layers.LSTM(500, return_sequences=True, activation="sigmoid"))  
model.add(layers.Dropout(0.2))  
model.add(layers.LSTM(300, return_sequences=False))  
model.add(layers.Dropout(0.2))  
model.add(layers.Dense(300, activation="relu"))  
model.add(layers.Activation("softmax"))  
model.build((None, 641, 300))
model.summary()
model.compile(optimizer="rmsprop", loss=tf.keras.losses.MeanSquaredError(), metrics=['accuracy'])

Is this proper usage of LSTM? The code runs fine, no errors, but the model simply doesn't train.

Improve this question
$\endgroup$
8
  • $\begingroup$ Many unusual things in your model: why so many LSTM layers? why a ReLU and then a softmax? Why a softmax at all? Are the vectors probability distributions? Why using dropout if your model does not work properly yet? $\endgroup$
    – noe
    Commented Nov 15, 2023 at 21:39
  • $\begingroup$ @noe To be honest, I'm not sure, my model architecture was taken from a tutorial that seems rather old. This is what the tutorial advised. Would you advise a single LSTM layer instead of multiple? Should relu and softmax not be used together? The vectors are text embeddings. $\endgroup$
    – slastine
    Commented Nov 15, 2023 at 21:41
  • $\begingroup$ Then there are many many wrong things in that code. I'd suggest following a different tutorial altogether. $\endgroup$
    – noe
    Commented Nov 15, 2023 at 21:46
  • $\begingroup$ @noe Probably good advice, but do you have any recommendations for improving the model in its current state? It's the only tutorial I can find on this topic. $\endgroup$
    – slastine
    Commented Nov 15, 2023 at 21:48
  • 1
    $\begingroup$ Sorry, that code has more wrong things than correct things. Some amendments for that code: start with a single LSTM, as input use the token indices and have an initial embedding layer, as output activation use softmax but without ReLU, use a categorical cross-entropy loss. Or better, check other examples, like this one for instance. $\endgroup$
    – noe
    Commented Nov 15, 2023 at 22:05

1 Answer 1

Reset to default
1
$\begingroup$

There are many problems with that code:

  • It uses a lot of LSTM layers. Instead, start with a single LSTM.
  • As you are working with text:
    • Use token indices as input and have an initial embedding layer
    • Use softmax (but without ReLU) as final activation.
    • Use token indices as expected output.
    • Use a categorical cross-entropy loss instead of MSE.

I suggest you use another tutorial. For instance, this notebook has the kind of setup that I was describing:

model1 = Sequential()
model1.add(Embedding(num_classes, embedding_size, input_length = maxlen))
model1.add(LSTM(lstm_size))
model1.add(Dense(num_classes, activation = 'softmax'))
model1.compile(loss = 'categorical_crossentropy', optimizer = 'adam')
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.