I have implemented my own mini neural network program1. Currently, it does not have batch updates, it only updates the parameters by simple backpropagation using SGD after each forward pass. I was trying to implement batch updates and batch normalisation2.
1) For simple batch updates, instead of updating parameters each time, for each image of the batch size of 'n' I should backpropagate and accumulate the deltas for all the parameters and finally update them once after the end of the batch.
2)For batch normalisation (BN), I went through the paper and I am sort of the clear with the idea but I am confused regarding how to implement it. Generally, I would multiply the matrices in the net one after the other for a single image to get final input, but with BN, do I need to feed forward for all the images in the batch till the first layer, then normalise the values, then fwd pass these values till second layer, then normalise again, and so on? Once I reach the final layer, should I backpropagate the error for the corresponding input-output pair and update the parameters immediately as fwd pass for all the images in the batch has been done already?
Going by the way I have described, it seems to require a lot of parameter tracking throughout the batch. It will be helpful if you can point out a better way to do it or anything that I have misunderstood so far.