5
$\begingroup$

I have scoured the internet and books, but everything seems to use num_steps and batch_size or similar terms interchangeably and I can not get a grasp on their specific use, or where batches fit in when talking about RNNs ( I understand their use with Gradient Descent, and maybe I am misunderstanding RNNs and I need to be set straight at the fundamental level- if that is the case, please do so.

So where does batching apply in Recurrent Neural Networks? At the level of instances within time steps or at the level of timesteps themselves.

By my current understanding the data fed into the model is structured like this:

data = [
    "time_step_1": [
        "instance_1": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_2": [
        "instance_2_but_first_in_time_step_2": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_3": [
        "instance_3_but_first_in_time_step_3": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_4": [
        "instance_4_but_first_in_time_step_4": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_5": [
        "instance_5_but_first_in_time_step_5": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],

    ]            
]

Except: That many people (and examples) have more instances per time step.

A typical training loop looks like:

for epoch in range(do_num_epochs):
    for time_step_count in range(len(data)):
        run(model, data[time_step_count])

And that is if you had (for example) 10,000 instances at each time step, then batching would allow the weights to be updated more often than having to wait for the entire 10,000 instances to be processed. Then the loop would look like:

batch_size = len(data) // do_num_batches

for epoch in range(do_num_epochs):
    for time_step_count in range(len(data)):
        for batch_count in range(do_num_batches):
            batch_data = data[time_step_count][batch_count * batch_size : (batch_count + 1) * batch_size]
            run(model, batch_data)

But there is no point in processing the data in batches if there is only a single instance of data per time step is there?

I made a chart to try to explain ( to myself as well ) what the flow and data structure would look like:

Data Structure

Model

I am just trying to understand the role of the batch in all of this, how it fits in?

(Edit/Addition)

After some more research on the subject, I am fairly certain that batches are a way to feed in sequences in chunks, and control the frequency of weight updates. By utilizing batches, the weights are updated at batch level, thus letting you group time-series together in groups. I illustrated what I believe to be right below, in the below example there are:

  • 4 instances,
  • each with 5 features
  • across 30 time steps

enter image description here

In the first example (mini-batch), there are 3 batches, of batch_size= 10 in that example, the weights would be updated 3 times, once after the conclusion of each batch.

In the second example, is online learning with an effective batch_size=1 and in that example, the weights would be updated 30 times, once after each time_series

In the final example (Batch learning), the entire data set is processed at once, the weights are updated only once (1) at the conclusion of the 'batch'.

In each of these example, the entire data set would be run through many multiple of times, each one of these cycles is an epoch.

$\endgroup$
  • $\begingroup$ So, basically, batching in RNNs is not about grouping network evaluations together as with feedforward networks, but only about grouping weight updates together ? Am I right ? $\endgroup$ – dev1223 Sep 9 '17 at 20:42
  • $\begingroup$ Yes, I am not the authority on all things Data Science. But over the past month every piece of additional information has confirmed this. Hopefully that helped. $\endgroup$ – ehiller Sep 9 '17 at 20:53
  • $\begingroup$ Thank you very much, any intel about ML has price of gold. $\endgroup$ – dev1223 Sep 9 '17 at 21:12
  • $\begingroup$ can somebody point out in how to update weights after each batch? For eg : there are 10 samples per batch, do I add the updates, for the 10 samples and then add to the weights with learning rate at the end of the batch? OR should I consider the average or if there's another measurE? $\endgroup$ – Lakshmi Narayanan Oct 17 '17 at 12:12
3
$\begingroup$

Your edit is not right. from the keras documentation you can actually understand the difference between timesteps and batches. I take your examples:

For the first example. You have 4 instances, or samples, or sequences. The length of each sequence is 30, now you actually take 3 batches, each of these batch is composed by 4 instances, or samples, or subsequences, of length 10(timesteps). your batch size is not 10 but 4, the timesteps don't affect the batch size. the subsequences of each batch can have the length that you want, this doesn't change the fact that the batch size is 4.

For the second example is the same. Online learning doesn't mean that the number of timesteps of each subsequence is 1, you can take the number of timesteps you want, but when you update the parameters of the network you should consider only 1 subsequence at the time.

You can actually confirm this from here:

http://philipperemy.github.io/keras-stateful-lstm/

In this part:

"Said differently, whenever you train or test your LSTM, you first have to build your input matrix X of shape nb_samples, timesteps, input_dim where your batch size divides nb_samples. For instance, if nb_samples=1024 and batch_size=64, it means that your model will receive blocks of 64 samples, compute each output (whatever the number of timesteps is for every sample), average the gradients and propagate it to update the parameters vector."

So the timesteps don't affect the batch size.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.