Batch Size of Stateful LSTM in keras

Question

My Model is defined as below:

## defining the model
batch_size = 1

def my_model():

   
   input_x = Input(batch_shape=(batch_size, look_back, 4), name='input')
   drop = Dropout(0.5)
   
   lstm_1 = LSTM(100, return_sequences=True, batch_input_shape=(batch_size, look_back, 4), name='3dLSTM', stateful=True)(input_x)
   lstm_1_drop = drop(lstm_1)
   lstm_2 = LSTM(100, batch_input_shape=(batch_size, look_back, 4), name='2dLSTM', stateful=True)(lstm_1_drop)
   lstm_2_drop = drop(lstm_2)

   y1 = Dense(1, activation='relu', name='op1')(lstm_2_drop)
   y2 = Dense(1, activation='relu', name='op2')(lstm_2_drop)
   
   model = Model(inputs=input_x, outputs=[y1,y2])
   model.compile(loss='mse', optimizer='adam',metrics=['mse'])
   model.summary()
   return model

model = my_model()

It is a Stateful LSTM model with batch size =1. My model.fit looks like this :

# Train the model
history = model.fit(
    x_train,
    [y_11_train,y_22_train],
    epochs=1, 
    batch_size=batch_size, 
    verbose=0, 
    shuffle=False)

model.reset_states()

My model runs well and outputs results. But I am unable to increase or alter the batch_size as flexibly as we could do when the model is in stateless condition. As in for bigger size of dataset the model seems to be training forever as the batch_size here is just 1. And as we know we can't randomly put any batch_size value for stateful LSTM as it needs to be a divisible factor.

I have gone through some blogs which describes changing batch_size by using different batch sizes for training and predicting using the get_weights() and set_weights() functions in the Keras API ref: https://machinelearningmastery.com/use-different-batch-sizes-training-predicting-python-keras/, still it seems to be that the batch_size that is being used here is less.

My question is : Cant we use batch size like 32, 64, 128 in stateful LSTMs? If yes then how do I implement it in my above given model, if not then what are the alternatives?

Looking for valuable suggestions.

Post Edit

In stateful LSTM model.reset_states() should be after every epoch, Hence I set the resetting of the states after each epoch in the following way:

for i in range(100):
    start = time.time()
    history = model.fit(x_train, [y_11_train,y_22_train], epochs=1, batch_size=batch_size, verbose=0, shuffle=False)
    model.reset_states()

    print("Epoch",i, time.time()-start,"s")

Answer 1

I solved the problem this way:

I realized that I needed to find the HCF (highest common factor) of both the length of x_train and x_test. For this I wrote a simple HCF python code the output of it is the batch_size.

Hence data set of different size will have different batch sizes.

Following is the code I used to find the HCF:

def computeHCF(x, y):
    if x > y:
        smaller = y
    else:
        smaller = x
    for i in range(1, smaller+1):
        if((x % i == 0) and (y % i == 0)):
            hcf = i

    return hcf

batch_size= computeHCF(x_train.shape[0], x_test.shape[0])

And use this batch_size both while defining the model, fitting the model(model.fit(..)) & predicting(model.predict(...)).

NOTE: We need to specify batch size while predicting the model. As:

model.predict(x_test, batch_size=batch_size)

Hope this helps someone!

Answer 2

The key part is, as you mentioned, batch size must be a value that divides without remainder into (I believe) the train and validation test set sizes.

One could find the highest common multiple (a.k.a. greatest common factor) of the dimensions of those two datasets, and that is your maximum batch size in a stateful LSTM. Check out this simple explanation if that is unclear.

You could try just a simple loop over something like:

for batch_size in range(128):
    try:
        model.train(...)
        print('Trained with batch size: {}'.format(batch_size))
    except:
        print('Couldn't use batch size: {}'.format(batch_size))

Just set #epochs to 1 and number of iterations to something small, or do whatever you can to reduce the time taken in each loop as you don't actually care about the results.