Batching in Recurrent Neural Networks (RNNs) when there is only a single instance per time step?

Question

I have scoured the internet and books, but everything seems to use num_steps and batch_size or similar terms interchangeably and I can not get a grasp on their specific use, or where batches fit in when talking about RNNs ( I understand their use with Gradient Descent, and maybe I am misunderstanding RNNs and I need to be set straight at the fundamental level- if that is the case, please do so.

So where does batching apply in Recurrent Neural Networks? At the level of instances within time steps or at the level of timesteps themselves.

By my current understanding the data fed into the model is structured like this:

data = [
    "time_step_1": [
        "instance_1": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_2": [
        "instance_2_but_first_in_time_step_2": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_3": [
        "instance_3_but_first_in_time_step_3": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_4": [
        "instance_4_but_first_in_time_step_4": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_5": [
        "instance_5_but_first_in_time_step_5": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],

    ]            
]

Except: That many people (and examples) have more instances per time step.

A typical training loop looks like:

for epoch in range(do_num_epochs):
    for time_step_count in range(len(data)):
        run(model, data[time_step_count])

And that is if you had (for example) 10,000 instances at each time step, then batching would allow the weights to be updated more often than having to wait for the entire 10,000 instances to be processed. Then the loop would look like:

batch_size = len(data) // do_num_batches

for epoch in range(do_num_epochs):
    for time_step_count in range(len(data)):
        for batch_count in range(do_num_batches):
            batch_data = data[time_step_count][batch_count * batch_size : (batch_count + 1) * batch_size]
            run(model, batch_data)

But there is no point in processing the data in batches if there is only a single instance of data per time step is there?

I made a chart to try to explain ( to myself as well ) what the flow and data structure would look like:

I am just trying to understand the role of the batch in all of this, how it fits in?

(Edit/Addition)

After some more research on the subject, I am fairly certain that batches are a way to feed in sequences in chunks, and control the frequency of weight updates. By utilizing batches, the weights are updated at batch level, thus letting you group time-series together in groups. I illustrated what I believe to be right below, in the below example there are:

• 4 instances,
• each with 5 features
• across 30 time steps

In the first example (mini-batch), there are 3 batches, of batch_size= 10 in that example, the weights would be updated 3 times, once after the conclusion of each batch.

In the second example, is online learning with an effective batch_size=1 and in that example, the weights would be updated 30 times, once after each time_series

In the final example (Batch learning), the entire data set is processed at once, the weights are updated only once (1) at the conclusion of the 'batch'.

In each of these example, the entire data set would be run through many multiple of times, each one of these cycles is an epoch.

Answer 1

Your edit is not right. from the keras documentation you can actually understand the difference between timesteps and batches. I take your examples:

For the first example. You have 4 instances, or samples, or sequences. The length of each sequence is 30, now you actually take 3 batches, each of these batch is composed by 4 instances, or samples, or subsequences, of length 10(timesteps). your batch size is not 10 but 4, the timesteps don't affect the batch size. the subsequences of each batch can have the length that you want, this doesn't change the fact that the batch size is 4.

For the second example is the same. Online learning doesn't mean that the number of timesteps of each subsequence is 1, you can take the number of timesteps you want, but when you update the parameters of the network you should consider only 1 subsequence at the time.

You can actually confirm this from here:

http://philipperemy.github.io/keras-stateful-lstm/

In this part:

"Said differently, whenever you train or test your LSTM, you first have to build your input matrix X of shape nb_samples, timesteps, input_dim where your batch size divides nb_samples. For instance, if nb_samples=1024 and batch_size=64, it means that your model will receive blocks of 64 samples, compute each output (whatever the number of timesteps is for every sample), average the gradients and propagate it to update the parameters vector."

So the timesteps don't affect the batch size.

Answer 2

I wanted to add a comment but due to my reputation I can only post an answer... @erre4 your example seems to contradict the github example you quoted:

• Your example: sequence length of an instance divided by the batch size
• Github example: total number of instances is divided by the batch size

The github example agrees with what I thought was the case, which is that batching does not alter the window size of a time series, or the number of features. Please clarify, thank you.