I can't understand why training of a neural network will require multiple iterations (theoretically)?
Can anyone explain why, please?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
Solving optimisation problems is difficult, and finding a closed-form solution that finds the optimal point for the cost function is complicated. Consequently, optimisation problems are solved using iterative steps. This means people choose solutions which are guaranteed to decrease the cost or objective function with each step. This idea is somehow used in neural networks.
Training a neural network involves optimizing a large set of parameters. This optimization step is commonly known as backpropgation (a.k.a backprop) via a form of gradient descent. Backprop via gradient descent allows a network to adjust its learnable parameters (i.e., weights) such that the loss (difference between the forward pass output and the actual output) reduces. In each epoch (1 pass over the entire dataset), very small adjustments (learning rate) are applied to the weights so that it eventually converges to an a a solution (a global optimum). However, finding a global optimum requires huge amounts of data. We never may know how much data we need until it is able to find a global optimum, therefore, in practice, we have multiple epochs (multiple passes over the data) to try to find this global optimum. In short, gradient descent is a very data-hungry process and hence why we require multiple epochs (or iteration in your case)
A neural network is usually trained on a large set of paired example data (supervised learning). For each example in the set, the best known optimization for each weight is calculated, but it is then multiplied by the learning rate, which is a very small number.
If a learning rate was not used, you would make large adjustments to each weight, only to destroy any positive benefit from those changes by making more large adjustments on the next example. By the time you reached the bottom of an iteration (epoch), your first few optimizations would be a distant memory, completely overwritten by later changes.
If we only make small adjustments to the weights after looking at each example, we can go over the training set again and again with the goal of achieving an average optimization for the whole set, not just any one example. This also helps to protect against overfitting, which is where the network learns the noise in the training data and hence performs great on the training set but poorly in the wild.