1
$\begingroup$

I am trying to use an LSTM model to make binary classifications; however when I train the model the loss stays around 0.69 (ie. -$\ln(0.5)$) and the accuracy at 0.5, which suggests to me the model is not learning as these are the numbers you would expect for a random guess. I have tried playing with the learning rate, changing the number of units, and stacking LSTM's together, but I feel that I am missing something about the use of state, but I'm not sure what.

My time series is of the form $t = [x_0, ..., x_N]$ and I wish to make use a rolling window of stride 10 to use 100 elements predictions about the sign of the next element. Eg. I want to us elements 0 to 99 to make a prediction about the 100th element's sign; then I want to use elements 10 to 109 to make predictions about the 110th etc.

Hence I have constructed my training set as

$X = \begin{bmatrix} x_{0} & x_{1} & x_{2} & \dots & x_{99} \\ x_{10} & x_{11} & x_{12} & \dots & x_{109} \\ ...\\ \end{bmatrix}$

and my target vector as, where a postive sign of x is 1 and a negative 0.

$y = \begin{bmatrix} \text{sgn}(x_{100}) \\ \text{sgn}(x_{110}) \\ \vdots \\ \end{bmatrix} $

In my case $N = 1047700$ so I have $10477$ sequences of length $100$. My idea is to use an LSTM with the following code

model = Sequential()

model.add(LSTM(32, activation='tanh',input_shape=(X.shape[1],1)))
model.add(Dense(1,activation='sigmoid'))
adm = Adam(lr=0.001)
model.compile(loss='binary_crossentropy', optimizer=adm, metrics=['accuracy'])

model.fit(train_array,classified_returns, epochs=200, verbose=1,
         shuffle=False, validation_split=0.1)

Here is the model summary

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_2 (LSTM)                (None, 32)                4608      
_________________________________________________________________
dense_13 (Dense)             (None, 1)                 33        
=================================================================
Total params: 4,641
Trainable params: 4,641
Non-trainable params: 0
_________________________________________________________________

Here is the extract output from fitting

Train on 9429 samples, validate on 1048 samples
Epoch 1/200
9429/9429 [==============================] - 18s 2ms/step - loss: 0.6932 - acc: 0.5002 - val_loss: 0.6931 - val_acc: 0.503849
Epoch 2/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6933 - acc: 0.4984 - val_loss: 0.6931 - val_acc: 0.5048
Epoch 3/200
9429/9429 [==============================] - 15s 2ms/step - loss: 0.6932 - acc: 0.4994 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 4/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4992 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 5/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4987 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 6/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4980 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 7/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4978 - val_loss: 0.6930 - val_acc: 0.5038
Epoch 8/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4984 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 9/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4986 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 10/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4977 - val_loss: 0.6930 - val_acc: 0.5057
Epoch 11/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4980 - val_loss: 0.6930 - val_acc: 0.5057
Epoch 12/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4981 - val_loss: 0.6930 - val_acc: 0.5067
Epoch 13/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4987 - val_loss: 0.6930 - val_acc: 0.5076
Epoch 14/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4978 - val_loss: 0.6930 - val_acc: 0.5076
Epoch 15/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4974 - val_loss: 0.6930 - val_acc: 0.5076
Epoch 16/200
9429/9429 [==============================] - 15s 2ms/step - loss: 0.6932 - acc: 0.4976 - val_loss: 0.6930 - val_acc: 0.5076
Epoch 17/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4968 - val_loss: 0.6930 - val_acc: 0.5076
Epoch 18/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4971 - val_loss: 0.6930 - val_acc: 0.5076

As we can see the model is not learning. Can someone point me in the right direction with statefulness?

$\endgroup$
0
$\begingroup$

How to use stateful LSTMs is quite well documented in the official Keras documentation. There is als a nice blog post here.

There is a note in the Recurrent Layers section:

Note on using statefulness in RNNs

You can set RNN layers to be 'stateful', which means that the states computed for the samples in one batch will be reused as initial states for the samples in the next batch. This assumes a one-to-one mapping between samples in different successive batches.

To enable statefulness: - specify stateful=True in the layer constructor. - specify a fixed batch size for your model, by passing if sequential model: batch_input_shape=(...) to the first layer in your model. else for functional model with 1 or more Input layers: batch_shape=(...) to all the first layers in your model. This is the expected shape of your inputs including the batch size. It should be a tuple of integers, e.g. (32, 10, 100). - specify shuffle=False when calling fit().

To reset the states of your model, call .reset_states() on either a specific layer, or on your entire model.

Small gotcha

The small snag that you have to fix yourself is to make sure your number of samples is an integer multiple of yur batch size: num_samples % batch_size == 0. You need to ensure that maanually because Keras will just take remaining samples for the final batch and this is usually smaller than you desired batch size.

$\endgroup$
7
  • $\begingroup$ I read this blog post, but he only provides an example with batch size = 1. Should I use the length of a sequence as a batch size? I tried using his code where he uses a batch size 1 and then resets state after the length of his sequence, but that does not improve learning. $\endgroup$
    – mch56
    May 2 '19 at 11:59
  • $\begingroup$ It is important that the batch size is constant. You can just use batch_input_shape t specify this (as in the Keras Documentation above). Please see my edit for extra info. $\endgroup$
    – n1k31t4
    May 2 '19 at 12:04
  • $\begingroup$ Ok thanks. Is resetting the state between batches the right thing to do to get this to learn? $\endgroup$
    – mch56
    May 2 '19 at 12:09
  • $\begingroup$ It will depend on your data and I'd suggest trying with and without. In general however, stateful LSTMs are used when you expect a relationship between batches (as in time-series). That's also why we set shuffle=False. If you reset the states, you are essentially removing the statefulness between batches! $\endgroup$
    – n1k31t4
    May 2 '19 at 12:13
  • $\begingroup$ So if I want to retain state between batches I need to set stateful=True and then use my entire input set as a the batch size, ie. 10477 in my case? This causes problems for the validation split though right? $\endgroup$
    – mch56
    May 2 '19 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.