In CNNs when do we update the kernel parameters using back propagation? Suppose I have batch size of 50 and training data of 1000. Do I back propagate after each batch has been presented to network or after each data sample?
Whenever you train the network using batch means that you have chosen to train using batch gradient descent. There are three variants for gradient descent algorithm:
- Gradient Descent
- Stochastic Gradient Descent
- Batch Gradient Descent
The first one passes the whole data through the network and finds the error rate for all of them and finds the gradients with respect to all the data samples and updates the weights after passing the whole data-set. That means for each epoch, passing the whole data-set through the network, one update occurs. This update is accurate toward descending gradient.
The second one, updates the weights after passing each data which means if your data sample has one thousand samples, one thousand updates will happen whilst the previous method updates the weights one time per the whole data-sample. This method is not accurate but is so much faster than the previous one.
The last one tries to find a trade-off between the above approaches. You specify a batch size and you will update the weights after passing the data samples in each batch, means the gradients are calculated after passing each batch. Suppose you have one thousand data sample and you have specified a batch size with one hundred data sample. You will have 10 weight update for each epoch. This method is more accurate than the second approach and is more faster than the first approach.
Do I back propagate after each batch has been presented to network or after each image?
Your method is the last one. Consequently, after passing the entire batch, you would update the weights.
Based on the comments of one of our friends, the above approaches are named as follows, respectively:
- [Batch] Gradient Descent (batch size = all training samples)
- True SGD (batch size = 1 - weights update for each training sample)
- Mini-batch SGD (batch size = m out of n training samples).
Back-propagation technically refers to computing the gradient of the loss function with respect to the parameters. According to Section 6.5 of the Deep Learning book:
The term back-propagation is often misunderstood as meaning the whole learning algorithm for multi layer neural networks. Actually, back-propagation refers only to the method for computing the gradient, while another algorithm,such as stochastic gradient descent, is used to perform learning using this gradient.
The weights are updated right after back-propagation in each iteration of stochastic gradient descent. From Section 8.3.1:
Here you can see that the parameters are updated by multiplying the gradient by the learning rate and subtracting.
The SGD algorithm described here applies to CNNs as well as other architectures.
Weight update can be understood as a change in weight to make your error less and less. YOu first assume some weights and get the model prediction and then the error. You then take the derivate of the error wrt to weights and finally update the weights so the error will reduce. you can also see it as a minimum finding problem in calculus. you are finding the values of weight or say the point in weight space that gives you the minimum error.
During The training stage, it just takes input and assigns values in a matrix. When you start giving it test data during the first iteration it just choose some random weights and try to figure out what it gets and then take that prediction as 'i' value based on the prediction, if it predicts true it doesn't change any of the weights for that class but if it predicted wrong class then it adjust weights according to parameters you have given then iterate again and until the prediction close to data what it is trained on.So, here only while test your data weights are altered and back propagated it may happen in the background by splitting your data of "1000" into training-set & test-set and once the training is completed test data is iterated multiple times until the loss stabilizes to certain point it may be different for each dataset your training on