1
$\begingroup$

I am working on classification problem, My input data is labels and output expected data is labels I have made X, Y pairs by shifting the X and Y is changed to the categorical value

Labels  Count
1      94481
0      65181
2      60448
    X   Y
    2   1.0
    1   2.0
    1   1.0
    2   1.0
    2   2.0
encoder = LabelEncoder()
test_labels = to_categorical(encoder.fit_transform(values[:,1]),num_classes=3)
train_X,test_X,train_y,test_y= train_test_split(values[:,0], test_labels,test_size = 0.30,random_state = 42)

print(train_X.shape)
print(train_y.shape)
print(test_X.shape)
print(test_y.shape)

(154076,) (154076, 3) (66033,) (66033, 3)
Converting this to LSTM format

train_X = train_X.reshape(train_X.shape[0],1,1)
test_X = test_X.reshape(test_X.shape[0],1,1)

# configure network
n_batch = 1
n_epoch = 10
n_neurons = 100

ModelArchitecture

tf.keras.backend.clear_session()

model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(n_neurons, batch_input_shape=(n_batch, train_X.shape[1],train_X.shape[2]), stateful=True),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(100, activation = 'relu',kernel_regularizer=regularizers.l2(0.0001)),
    tf.keras.layers.Dense(3, activation='softmax')
])

model.summary()
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['acc'])
history = model.fit(train_X,train_y,validation_data=(test_X, test_y),epochs=n_epoch, batch_size=n_batch, verbose=1,shuffle= False)

Validation Accuracy is not Changing

Epoch 1/5
154076/154076 [==============================] - 356s 2ms/step - loss: 1.0844 - acc: 0.4269 - val_loss: 1.0814 - val_acc: 0.4310
Epoch 2/5
154076/154076 [==============================] - 354s 2ms/step - loss: 1.0853 - acc: 0.4256 - val_loss: 1.0813 - val_acc: 0.4310
Epoch 3/5
154076/154076 [==============================] - 355s 2ms/step - loss: 1.0861 - acc: 0.4246 - val_loss: 1.0814 - val_acc: 0.4310
Epoch 4/5
154076/154076 [==============================] - 356s 2ms/step - loss: 1.0874 - acc: 0.4228 - val_loss: 1.0825 - val_acc: 0.4310
Epoch 5/5
154076/154076 [==============================] - 353s 2ms/step - loss: 1.0887 - acc: 0.4208 - val_loss: 1.0828 - val_acc: 0.4310

What can be the changes to improve the model.

Update2

After Increasing the learning rate of Rmsprop to 0.5 , Below is the training loss and validation loss

Epoch 1/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.4314 - acc: 0.3594 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 2/10
176087/176087 [==============================] - 369s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 3/10
176087/176087 [==============================] - 371s 2ms/step - loss: 1.3209 - acc: 0.3592 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 4/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.3209 - acc: 0.3592 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 5/10
176087/176087 [==============================] - 368s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 6/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 7/10
176087/176087 [==============================] - 368s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 8/10
176087/176087 [==============================] - 372s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 9/10
176087/176087 [==============================] - 371s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9379 - val_acc: 0.2739
Epoch 10/10
176087/176087 [==============================] - 369s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
$\endgroup$
4
  • $\begingroup$ Have you tried increasing the learning rate? It looks like your training loss isn't changing $\endgroup$ Jun 28 '20 at 12:55
  • $\begingroup$ @DavidMasip I have changed the learning rate and it clearing indicating me of overfitting as i can see the training loss is very much lesser than validation loss $\endgroup$ Jun 28 '20 at 14:04
  • $\begingroup$ Can you share the history variable in this case? The evolution of the loss, I mean $\endgroup$ Jun 28 '20 at 14:43
  • $\begingroup$ @DavidMasip please check the update2 and let me know your observation $\endgroup$ Jun 28 '20 at 15:05
0
$\begingroup$

From the documentation: inputs: A 3D tensor with shape [batch, timesteps, feature].

Are you saying that you want 1 input and 1 feature, but you want to output 100 neurons? I would consider adding more timesteps.

train_X = train_X.reshape(train_X.shape[0],10,1)
test_X = test_X.reshape(test_X.shape[0],10,1)

Also, I wouldn't add regularization to a ReLU activation without batch normalization. Actually, I probably would use dropout instead of regularization. Consider that with regularization many ReLU neurons may die.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.