13
$\begingroup$

My Model is defined as below:

## defining the model
batch_size = 1

def my_model():

   
   input_x = Input(batch_shape=(batch_size, look_back, 4), name='input')
   drop = Dropout(0.5)
   
   lstm_1 = LSTM(100, return_sequences=True, batch_input_shape=(batch_size, look_back, 4), name='3dLSTM', stateful=True)(input_x)
   lstm_1_drop = drop(lstm_1)
   lstm_2 = LSTM(100, batch_input_shape=(batch_size, look_back, 4), name='2dLSTM', stateful=True)(lstm_1_drop)
   lstm_2_drop = drop(lstm_2)

   y1 = Dense(1, activation='relu', name='op1')(lstm_2_drop)
   y2 = Dense(1, activation='relu', name='op2')(lstm_2_drop)
   
   model = Model(inputs=input_x, outputs=[y1,y2])
   model.compile(loss='mse', optimizer='adam',metrics=['mse'])
   model.summary()
   return model

model = my_model()

It is a Stateful LSTM model with batch size =1. My model.fit looks like this :

# Train the model
history = model.fit(
    x_train,
    [y_11_train,y_22_train],
    epochs=1, 
    batch_size=batch_size, 
    verbose=0, 
    shuffle=False)

model.reset_states()

My model runs well and outputs results. But I am unable to increase or alter the batch_size as flexibly as we could do when the model is in stateless condition. As in for bigger size of dataset the model seems to be training forever as the batch_size here is just 1. And as we know we can't randomly put any batch_size value for stateful LSTM as it needs to be a divisible factor.

I have gone through some blogs which describes changing batch_size by using different batch sizes for training and predicting using the get_weights() and set_weights() functions in the Keras API ref: https://machinelearningmastery.com/use-different-batch-sizes-training-predicting-python-keras/, still it seems to be that the batch_size that is being used here is less.

My question is : Cant we use batch size like 32, 64, 128 in stateful LSTMs? If yes then how do I implement it in my above given model, if not then what are the alternatives?

Looking for valuable suggestions.

Post Edit

In stateful LSTM model.reset_states() should be after every epoch, Hence I set the resetting of the states after each epoch in the following way:

for i in range(100):
    start = time.time()
    history = model.fit(x_train, [y_11_train,y_22_train], epochs=1, batch_size=batch_size, verbose=0, shuffle=False)
    model.reset_states()

    print("Epoch",i, time.time()-start,"s")
$\endgroup$
1
  • $\begingroup$ I may be missing something but I believe you can just set batch size to None $\endgroup$
    – kbrose
    Aug 1, 2018 at 12:56

2 Answers 2

10
$\begingroup$

I solved the problem this way:

I realized that I needed to find the HCF (highest common factor) of both the length of x_train and x_test. For this I wrote a simple HCF python code the output of it is the batch_size.

Hence data set of different size will have different batch sizes.

Following is the code I used to find the HCF:

def computeHCF(x, y):
    if x > y:
        smaller = y
    else:
        smaller = x
    for i in range(1, smaller+1):
        if((x % i == 0) and (y % i == 0)):
            hcf = i

    return hcf

batch_size= computeHCF(x_train.shape[0], x_test.shape[0])

And use this batch_size both while defining the model, fitting the model(model.fit(..)) & predicting(model.predict(...)).

NOTE: We need to specify batch size while predicting the model. As:

model.predict(x_test, batch_size=batch_size)

Hope this helps someone!

$\endgroup$
5
$\begingroup$

The key part is, as you mentioned, batch size must be a value that divides without remainder into (I believe) the train and validation test set sizes.

One could find the highest common multiple (a.k.a. greatest common factor) of the dimensions of those two datasets, and that is your maximum batch size in a stateful LSTM. Check out this simple explanation if that is unclear.

You could try just a simple loop over something like:

for batch_size in range(128):
    try:
        model.train(...)
        print('Trained with batch size: {}'.format(batch_size))
    except:
        print('Couldn't use batch size: {}'.format(batch_size))

Just set #epochs to 1 and number of iterations to something small, or do whatever you can to reduce the time taken in each loop as you don't actually care about the results.

$\endgroup$
3
  • $\begingroup$ @n1k31t4- Thanks, I took the idea of HCF from your post and made an answer. :) $\endgroup$
    – Jazz
    Jun 11, 2018 at 13:28
  • $\begingroup$ @n1k31t4 I dont understand why this should be the case. According to documentation, "stateful: If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.". So why should the size of the batch make a difference? $\endgroup$ Jan 16, 2019 at 23:11
  • $\begingroup$ @DankMasterDan - if you use batch_size=10 and have 101 samples, by default, Keras will send 10 batches of 10 samples and a final batch of 1 sample. That means a stateful layer will only receive 1 input at some point, when it is expecting 10. I suppose it should be possible to just ignore 9 indices, but it seems not to be implemented that way (or we are both missing some information!). There are useful examples here. $\endgroup$
    – n1k31t4
    Jan 16, 2019 at 23:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.