Training with multi-series of different length with stateful LSTM

Question

I'm training a stateful LSTM. My data is stored in a series of files, each file relates to a certain city. For each city I might have different amount of data, so City A I might have 4000 days, but City B I might only have 500. I have 20 features for each day.

Say I set the sequence size to 200; for City A I'd have a batch of 20 complete sequences, but for City B I'd only have 2.

With stateful LSTM I need to specify the batch_input_shape in the form [batch size, sequence size, features], which I think will cause me some problems because my batch size changes for each city I process.

As I see it, the only option available to me is to define the batch size for batch_input_shape as 1, and then just effectively issue a single 200-length sequence for each iteration. So batch_input_shape would be [1, 200, 20]

Obviously this isn't ideal because it's such a small amount of data that's being presented to the NN for each iteration (it's certainly mini-batch!) so it will be quite slow to train the network.

But am I right that due to the differing sizes of the data, I don't really have another option? Besides slow train time, are there any inherent problems training with batch size of 1?

Answer 1

To begin with, you can think of the batch size as a way to control the smoothness of the learning curve. With a huge batch size, you are taking the average of many errors for each update, and this average loss (on average), doesn't have great variance.

Using a batch size of 1, your cost on each iteration is solely dependent on the single sample that you fed the network. Each sample is hopefully a little different from the others, which will lead to a very noisy loss curve.

Assuming model parameters are well-suited and the model converges irrespective of batch size, you should reach similar results. However, there are works that analyse batch size and show other trade-off, which are usually training-times, memory consumption and the like.

This recent paper (Masters, Luschi) analyses the trade-offs of batch size on a standard dataset (CIFAR10). Here is Figure 15 from the paper that shows this pretty succinctly:

Warning: all models and datasets could behave differently, however, so don't take any results like these as fact!

---

I know that it is possible to train a model that has varying input shapes by using a technique called Adaptive Pooling (or adaptive average pooling, in PyTorch, but you would likely have to come up with your own function that is able to do such a thing within the constraints of a stateful LSTM.

So as the shape of your dataset must be divisible by the batch size, there are a few ways to make that a reality:

1. using the highest common multiple of your datasets: so with your example datasets having 4000 days (set A) and 500 days (set B), you would compute the

This of course limits the possible choices of sequence length, but for the two examples of 4000 and 500, you could choose from these: 1, 2, 4, 5, 10, 20, 25, 50, 100

2. Trimming the different datasets so that you get a nice sequence length that works for both

This means possible leaving out some of your data, which is undesireable, but it might not be much.

Which of those two possiblities you might go for will depend on your specific dataset sizes. It might be optimal to use both - so trimming a few datasets to allow the computation of a nice Highest Common Factor.

You can compute the HCF in Python using something like this:

def hcf(a, b):
    while b:
        a, b = b, b % a
    return a

---

One final idea from my comment below:

... in order to change the batch size between datasets, you could copy them model weights from a trained model (from your first dataset), then compile a new model with the batch size required for the next dataset, but set the weights equal to those from the first model. This is almost like a manual way of implementing stateful behaviour. Have a look here for a simple example.