2
$\begingroup$

I have implemented my own mini neural network program1. Currently, it does not have batch updates, it only updates the parameters by simple backpropagation using SGD after each forward pass. I was trying to implement batch updates and batch normalisation2.

1) For simple batch updates, instead of updating parameters each time, for each image of the batch size of 'n' I should backpropagate and accumulate the deltas for all the parameters and finally update them once after the end of the batch.

2)For batch normalisation (BN), I went through the paper and I am sort of the clear with the idea but I am confused regarding how to implement it. Generally, I would multiply the matrices in the net one after the other for a single image to get final input, but with BN, do I need to feed forward for all the images in the batch till the first layer, then normalise the values, then fwd pass these values till second layer, then normalise again, and so on? Once I reach the final layer, should I backpropagate the error for the corresponding input-output pair and update the parameters immediately as fwd pass for all the images in the batch has been done already?

Going by the way I have described, it seems to require a lot of parameter tracking throughout the batch. It will be helpful if you can point out a better way to do it or anything that I have misunderstood so far.

$\endgroup$
1
$\begingroup$

Give the whole batch as an input, take the activated output of the first layer and pipe it the next until the final. .

calculate the gradient decent direction (derivative of the cost function) for each row. U will have the error for each row of ur training set.

Then add the errors and divide the result with number of the training set. Then finally update it weights. I know from experience that tracking all these numbers is hard.

I don't know what language u use, but if it's Python, numpy will help u a a lot. For example, to add the errors, errors.sum(axis=0). X.dot(weight) to multiply ur entire dataset with ur weight. So my suggestion is use some matrix library.

One thing I want to mention is batch processing is susceptible to local minima which u don't want...especially in complex functions. SDG is a nice algorithm. I'm not sure why u needed batch update. There are many interesting tweaks in NN, don't waste ur time on batch update, use SDG

$\endgroup$
  • $\begingroup$ Thanks @AmanuelNegash! Yeah, there's a lot of talk about batch updates getting stuck in local minima but what I figured out was that when training data is less then it's advantageous to stick with online (non-batch) SGD, otherwise batch updates do help for large datasets. Moreover, I wanted to implement BATCH NORMALISATION which builds upon batch updates but I was bit confused regarding it's implementation. Links which I have been following: Practical recommendation for SGD by Y.Bengio and Batch Normalization $\endgroup$ – Yash chandak Mar 23 '16 at 15:31
  • $\begingroup$ @Yashchandak That's right. May I suggest courses? Yes, coursera Geoffrey Hinton, neural net, udacuty Deep learning. They will give u ideas on how u can integrate tweaks to get best results. $\endgroup$ – Amanuel Negash Mar 23 '16 at 17:12
  • $\begingroup$ yeah, I have seen those, i personally prefer lectures by Hugo Larochelle, Montreal, i find them mathematically more rigorous. Normal batch updates are fine but the thing is they all cover (if at all) 'batch normalisation' very superficially. I had doubts regarding implementing this. I thought someone who has implemented it can pass on their two cents. Meanwhile, I will try looking into popular libraries for it's implementation. Thanks for the help anyway @AmanuelNegash! $\endgroup$ – Yash chandak Mar 24 '16 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.