6
$\begingroup$

I'm training a stateful LSTM. My data is stored in a series of files, each file relates to a certain city. For each city I might have different amount of data, so City A I might have 4000 days, but City B I might only have 500. I have 20 features for each day.

Say I set the sequence size to 200; for City A I'd have a batch of 20 complete sequences, but for City B I'd only have 2.

With stateful LSTM I need to specify the batch_input_shape in the form [batch size, sequence size, features], which I think will cause me some problems because my batch size changes for each city I process.

As I see it, the only option available to me is to define the batch size for batch_input_shape as 1, and then just effectively issue a single 200-length sequence for each iteration. So batch_input_shape would be [1, 200, 20]

Obviously this isn't ideal because it's such a small amount of data that's being presented to the NN for each iteration (it's certainly mini-batch!) so it will be quite slow to train the network.

But am I right that due to the differing sizes of the data, I don't really have another option? Besides slow train time, are there any inherent problems training with batch size of 1?

$\endgroup$
5
$\begingroup$

To begin with, you can think of the batch size as a way to control the smoothness of the learning curve. With a huge batch size, you are taking the average of many errors for each update, and this average loss (on average), doesn't have great variance.

Using a batch size of 1, your cost on each iteration is solely dependent on the single sample that you fed the network. Each sample is hopefully a little different from the others, which will lead to a very noisy loss curve.

Assuming model parameters are well-suited and the model converges irrespective of batch size, you should reach similar results. However, there are works that analyse batch size and show other trade-off, which are usually training-times, memory consumption and the like.

This recent paper (Masters, Luschi) analyses the trade-offs of batch size on a standard dataset (CIFAR10). Here is Figure 15 from the paper that shows this pretty succinctly:

performance based on batch-size

Warning: all models and datasets could behave differently, however, so don't take any results like these as fact!


I know that it is possible to train a model that has varying input shapes by using a technique called Adaptive Pooling (or adaptive average pooling, in PyTorch, but you would likely have to come up with your own function that is able to do such a thing within the constraints of a stateful LSTM.

So as the shape of your dataset must be divisible by the batch size, there are a few ways to make that a reality:

  1. using the highest common multiple of your datasets: so with your example datasets having 4000 days (set A) and 500 days (set B), you would compute the

This of course limits the possible choices of sequence length, but for the two examples of 4000 and 500, you could choose from these: 1, 2, 4, 5, 10, 20, 25, 50, 100

  1. Trimming the different datasets so that you get a nice sequence length that works for both

This means possible leaving out some of your data, which is undesireable, but it might not be much.

Which of those two possiblities you might go for will depend on your specific dataset sizes. It might be optimal to use both - so trimming a few datasets to allow the computation of a nice Highest Common Factor.

You can compute the HCF in Python using something like this:

def hcf(a, b):
    while b:
        a, b = b, b % a
    return a

One final idea from my comment below:

... in order to change the batch size between datasets, you could copy them model weights from a trained model (from your first dataset), then compile a new model with the batch size required for the next dataset, but set the weights equal to those from the first model. This is almost like a manual way of implementing stateful behaviour. Have a look here for a simple example.

$\endgroup$
  • $\begingroup$ Thanks for your reply, it is appreciated. The question I was most looking for help/confirmation of was whether my thought about needing to set batch size to 1 was correct due to my circumstances, or if there was some other way I would do it. The key point here being that this is stateful LSTM. But thank you for your input on having mini batch size 1 $\endgroup$ – BigBadMe Jul 8 '18 at 8:31
  • $\begingroup$ @BigMeat - Please see the additional info I just added. $\endgroup$ – n1k31t4 Jul 8 '18 at 12:23
  • $\begingroup$ Thanks for the additional info. With regard 1) would the resulting number of batches cause an issue? E.g. if I had sequence length 100, then one batch_input_shape for City A would be [400,100,20] and for City B it would be [5,100,20] Since the shapes differ wouldn't that cause an error as I must define when the model is created what the shape of the input will be? 2) It's a possibility, but there's such a varying amount of size I'd rather not lose (quite a lot of) data. I think I'd rather have batch size of 1 and be able to use all data, than lose data. $\endgroup$ – BigBadMe Jul 8 '18 at 16:43
  • 1
    $\begingroup$ @BigMeat - Just an idea: in order to change the batch size between datasets, you could copy them model weights from a trained model (from your first dataset), then compile a new model with the batch size required for the next dataset, but set the weights equal to those from the first model. This is almost like a manual way of implementing stateful behaviour. Have a look here for a simple example. Regarding 2), it is always a trade-off between training times, test accuracy and memory footprints. Just do the best you can with your time, data and hardware constraints. $\endgroup$ – n1k31t4 Jul 8 '18 at 19:52
  • $\begingroup$ Ahh yes, that's an idea. I guess I'd need to do some testing whether it's faster to have batch size = 1 or to copy/compile a new model for city I process but be able to provide bigger batches. Thanks for the suggestion. $\endgroup$ – BigBadMe Jul 9 '18 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.