# Using LSTM to predict binary classification - accuracy stuck at 50% - how to use statefulness

I am trying to use an LSTM model to make binary classifications; however when I train the model the loss stays around 0.69 (ie. -$$\ln(0.5)$$) and the accuracy at 0.5, which suggests to me the model is not learning as these are the numbers you would expect for a random guess. I have tried playing with the learning rate, changing the number of units, and stacking LSTM's together, but I feel that I am missing something about the use of state, but I'm not sure what.

My time series is of the form $$t = [x_0, ..., x_N]$$ and I wish to make use a rolling window of stride 10 to use 100 elements predictions about the sign of the next element. Eg. I want to us elements 0 to 99 to make a prediction about the 100th element's sign; then I want to use elements 10 to 109 to make predictions about the 110th etc.

Hence I have constructed my training set as

$$X = \begin{bmatrix} x_{0} & x_{1} & x_{2} & \dots & x_{99} \\ x_{10} & x_{11} & x_{12} & \dots & x_{109} \\ ...\\ \end{bmatrix}$$

and my target vector as, where a postive sign of x is 1 and a negative 0.

$$y = \begin{bmatrix} \text{sgn}(x_{100}) \\ \text{sgn}(x_{110}) \\ \vdots \\ \end{bmatrix}$$

In my case $$N = 1047700$$ so I have $$10477$$ sequences of length $$100$$. My idea is to use an LSTM with the following code

model = Sequential()

model.fit(train_array,classified_returns, epochs=200, verbose=1,
shuffle=False, validation_split=0.1)


Here is the model summary

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm_2 (LSTM)                (None, 32)                4608
_________________________________________________________________
dense_13 (Dense)             (None, 1)                 33
=================================================================
Total params: 4,641
Trainable params: 4,641
Non-trainable params: 0
_________________________________________________________________


Here is the extract output from fitting

Train on 9429 samples, validate on 1048 samples
Epoch 1/200
9429/9429 [==============================] - 18s 2ms/step - loss: 0.6932 - acc: 0.5002 - val_loss: 0.6931 - val_acc: 0.503849
Epoch 2/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6933 - acc: 0.4984 - val_loss: 0.6931 - val_acc: 0.5048
Epoch 3/200
9429/9429 [==============================] - 15s 2ms/step - loss: 0.6932 - acc: 0.4994 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 4/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4992 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 5/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4987 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 6/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4980 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 7/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4978 - val_loss: 0.6930 - val_acc: 0.5038
Epoch 8/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4984 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 9/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4986 - val_loss: 0.6930 - val_acc: 0.5048
Epoch 10/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4977 - val_loss: 0.6930 - val_acc: 0.5057
Epoch 11/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4980 - val_loss: 0.6930 - val_acc: 0.5057
Epoch 12/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4981 - val_loss: 0.6930 - val_acc: 0.5067
Epoch 13/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4987 - val_loss: 0.6930 - val_acc: 0.5076
Epoch 14/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4978 - val_loss: 0.6930 - val_acc: 0.5076
Epoch 15/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4974 - val_loss: 0.6930 - val_acc: 0.5076
Epoch 16/200
9429/9429 [==============================] - 15s 2ms/step - loss: 0.6932 - acc: 0.4976 - val_loss: 0.6930 - val_acc: 0.5076
Epoch 17/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4968 - val_loss: 0.6930 - val_acc: 0.5076
Epoch 18/200
9429/9429 [==============================] - 16s 2ms/step - loss: 0.6932 - acc: 0.4971 - val_loss: 0.6930 - val_acc: 0.5076


As we can see the model is not learning. Can someone point me in the right direction with statefulness?

• try hyperparameter tuning to find correct rolling window size, units, learning rate, batch size try modifying the input instead of number, change it to 0 or 1, 0 if value is less than previous day and otherwise, make lstm stateful, Commented Jul 11, 2022 at 6:50

How to use stateful LSTMs is quite well documented in the official Keras documentation. There is als a nice blog post here.

There is a note in the Recurrent Layers section:

Note on using statefulness in RNNs

You can set RNN layers to be 'stateful', which means that the states computed for the samples in one batch will be reused as initial states for the samples in the next batch. This assumes a one-to-one mapping between samples in different successive batches.

To enable statefulness: - specify stateful=True in the layer constructor. - specify a fixed batch size for your model, by passing if sequential model: batch_input_shape=(...) to the first layer in your model. else for functional model with 1 or more Input layers: batch_shape=(...) to all the first layers in your model. This is the expected shape of your inputs including the batch size. It should be a tuple of integers, e.g. (32, 10, 100). - specify shuffle=False when calling fit().

To reset the states of your model, call .reset_states() on either a specific layer, or on your entire model.

### Small gotcha

The small snag that you have to fix yourself is to make sure your number of samples is an integer multiple of yur batch size: num_samples % batch_size == 0. You need to ensure that maanually because Keras will just take remaining samples for the final batch and this is usually smaller than you desired batch size.

• I read this blog post, but he only provides an example with batch size = 1. Should I use the length of a sequence as a batch size? I tried using his code where he uses a batch size 1 and then resets state after the length of his sequence, but that does not improve learning. Commented May 2, 2019 at 11:59
• It is important that the batch size is constant. You can just use batch_input_shape t specify this (as in the Keras Documentation above). Please see my edit for extra info. Commented May 2, 2019 at 12:04
• Ok thanks. Is resetting the state between batches the right thing to do to get this to learn? Commented May 2, 2019 at 12:09
• It will depend on your data and I'd suggest trying with and without. In general however, stateful LSTMs are used when you expect a relationship between batches (as in time-series). That's also why we set shuffle=False. If you reset the states, you are essentially removing the statefulness between batches! Commented May 2, 2019 at 12:13
• Things tk try: using longer sequences, normalising your data, by either scaling to [- 1, 1] or just take the logarithm, add more layers then reset states on early layers to keep them more general. Use another model, e.g. ARIMA. it is commonly used for time series analysis and you will find a lot of material online to help. Commented May 2, 2019 at 13:02