-1
$\begingroup$

I can't understand why training of a neural network will require multiple iterations (theoretically)?

Can anyone explain why, please?

$\endgroup$
0
$\begingroup$

Solving optimisation problems is difficult, and finding a closed-form solution that finds the optimal point for the cost function is complicated. Consequently, optimisation problems are solved using iterative steps. This means people choose solutions which are guaranteed to decrease the cost or objective function with each step. This idea is somehow used in neural networks.

| improve this answer | |
$\endgroup$
  • $\begingroup$ You mean it's a optimization solution. So, can we have a non-iterative solution in NN at all? $\endgroup$ – Ali Aug 5 at 19:01
  • $\begingroup$ There are studies which have employed approaches like Genetic, but as far as I know, the prominent solutions are weight updatings by virtue of the current iterative methods. The point is that the cost function depends on numerous variables, and it realy does not have a closed form solution; you have no other choice, and the current iterative tools are the best. $\endgroup$ – Media Aug 5 at 19:56
0
$\begingroup$

Training a neural network involves optimizing a large set of parameters. This optimization step is commonly known as backpropgation (a.k.a backprop) via a form of gradient descent. Backprop via gradient descent allows a network to adjust its learnable parameters (i.e., weights) such that the loss (difference between the forward pass output and the actual output) reduces. In each epoch (1 pass over the entire dataset), very small adjustments (learning rate) are applied to the weights so that it eventually converges to an a a solution (a global optimum). However, finding a global optimum requires huge amounts of data. We never may know how much data we need until it is able to find a global optimum, therefore, in practice, we have multiple epochs (multiple passes over the data) to try to find this global optimum. In short, gradient descent is a very data-hungry process and hence why we require multiple epochs (or iteration in your case)

| improve this answer | |
$\endgroup$
0
$\begingroup$

A neural network is usually trained on a large set of paired example data (supervised learning). For each example in the set, the best known optimization for each weight is calculated, but it is then multiplied by the learning rate, which is a very small number.

If a learning rate was not used, you would make large adjustments to each weight, only to destroy any positive benefit from those changes by making more large adjustments on the next example. By the time you reached the bottom of an iteration (epoch), your first few optimizations would be a distant memory, completely overwritten by later changes.

If we only make small adjustments to the weights after looking at each example, we can go over the training set again and again with the goal of achieving an average optimization for the whole set, not just any one example. This also helps to protect against overfitting, which is where the network learns the noise in the training data and hence performs great on the training set but poorly in the wild.

| improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.