3
$\begingroup$

I am a beginner with rnns, consider this sample code

from tensorflow import keras
import numpy as np

if __name__ == '__main__':
    model = keras.Sequential((
        keras.layers.SimpleRNN(5, activation="softmax", input_shape=(1, 3)),
    ))
    X = [
        [1, 2, 3],
        [4, 5, 6]
    ]
    y = [
        [1, 0, 0, 0, 0],
        [0, 1, 0, 0, 0]
    ]
    X = np.array(X)
    X = np.reshape(X, (2, 1, 3))
    y = np.array(y)
    # print(X)
    # print(y)
    print(model.summary())
    model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.01),
                 loss="categorical_crossentropy")
    model.fit(X, y, epochs=100)
    p = model.predict(X)
    print(p)
    p = list(np.squeeze(p))
    print(p)
    print(np.sum(p,axis=1))

I am using a simple rnn with batch size=2, 3 input features and 1 timestep,as the activation is softmax the last line prints [1,1] as the sum of predictions of a softmax is 1. But when when I change the layer from a SimpleRNN to

keras.layers.LSTM(5, activation="softmax", input_shape= 
                 (1,3),recurrent_activation="softmax")

The sum of predictions is no longer 1, why is that?

$\endgroup$
0
$\begingroup$

Softmax doesnt work as an LSTM activation. You have to add a dense layer using a softmax activation after the LSTM layer. I wouls suggest using another activation in the LSTM like relu.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.