how to optimize the weights of a neural net when feeding it with multiple training samples?

Question

My question is:

Let's say we have a 3 by 1 neural net similar to that one in the image (an input layer with 3 neurons and an output layer with one neuron and no hidden layer), I find no problem calculating the outputs of the forward and back propagation when feeding the neural net with one training sample (namely feature1, feature2, feature3 inputs) and I know exactly how my initial weights get optimized, the problem I find is when feeding the NN with multiple training inputs each time, here, I don't know exactly how the initial weights get optimized.

For example we have training inputs of 3 × 3 Matrix.

[[195, 90, 41],
[140, 50, 30],
[180, 85, 43]]

the first column is the height, 2nd: the weight, 3rd: shoe size, we feed the NN with the first row then the second and the third row. We know that to calculate the new weights when feeding the NN with one training sample we rely on this formula: New_weights = Initial_weights - learning_rate × (derivative of the loss function wrt the weights).

But when we feed the NN with more than one training example then which formula do we use ? Do we calculate the average of all dw (derivative of the loss function wrt the weights) or do we sum all of'em then multiply by the learning rate and substract them from the initial weights or what? I'm a bit confused here.

I would be grateful if any of you could explain how the initial weights get modified when feeding the NN with multiple training inputs.

Answer 1

But when we feed the NN with more than one training example then which formula do we use ? Do we calculate the average of all dw (derivative of the loss function wrt the weights) or do we sum all of'em then multiply by the learning rate and substract them from the initial weights or what ?

This is batch gradient optimisation, so you average the gradients over all training examples in your batch.

Side note: In principle, you could use several training epochs (= several iterations over your entire dataset) and update the weights once per epoch. However, for large datasets (like ImageNet), you divide your dataset into batches and update the weights once per batch. You could also further subdivide each batch into minibatches and update once per minibatch. The subdivision is entirely up to you and is kind of ad hoc, but a recent paper suggests that consistently higher performance can be obtained with small minibatches of size between 2 and 32. Also, your learning rate should be scaled appropriately depending on the batch size.

Answer 2

Do we calculate the average of all dw (derivative of the loss function wrt the weights) or do we sum all of'em then multiply by the learning rate and substract them from the initial weights or what ?

Does not matter if you do either. The formulae for weight update is:

new_weights = initial_weights - learning_rate * average(gradient_over_all_samples).

Although it is the thumb rule to calculate average but it will not matter since calculating average is just multiplying it by a scalar 1/n which can be factored out and multiplied with the learning rate. Thus, instead of explicitly calculating average just include it in the learning_rate will save you time, computation and code.

Only difference between SGD and batch learning that you are doing is that the update is made after the entire dataset is traversed, whereas in SGD update is made after single example is traversed.