1
$\begingroup$
  1. Across different epochs, which of the following is/are updated?

initial weights (initial ConvNet filter matrices, initial fully connected weights)

hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers...

  1. The lost function calculated from the last epoch appears to be the initial value of the lost function for the current epoch. Why?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

  1. You are updating your network parameters, that is, weights for fully connected layers, for the filters in the convolution operations, etc. The hyperparameters are fixed once you start training your network. Hyperparameters are not intrinsic to the learning process and is something that the practitioner should tune carefully with GridSearch, Bayesian Optimization and Cross-Validation techniques.

  2. You have just one loss function during training, and at each batch procesing you update your weights correcting your network and, at least theoretically, diminishing your loss function. So after the first epoch, you have reached a certain value, that will be update on the next epoch. Think as you are on the top of a mountain, and you are climbing down, to no get tired, you count 10 steps and rest a little, after 10 steps you are not on the top again, you are going down from where you stopped, right? That is an analogy (I think it is bad, but if you understand it is ok haha).

Improve this answer
$\endgroup$
6
  • $\begingroup$ 1. so validation datasets dont affect any hyper parameters? are initial weights all re-randomized in each epoch? $\endgroup$
    – feynman
    Mar 8, 2019 at 14:05
  • $\begingroup$ 2. if all initial weights are re-randomized in each epoch, doesnt the lost function also start from an initial high value? if all initial weights are re-randomized, the training in each epoch should b a new training, irrespective of the last epoch? $\endgroup$
    – feynman
    Mar 8, 2019 at 14:08
  • 1
    $\begingroup$ No, the weights are not re-randomized at each epoch. They are on the start training process only. The weights that the second epoch will update are the ones from the last epoch, and It do not start again. $\endgroup$
    – Victor Oliveira
    Mar 8, 2019 at 14:10
  • $\begingroup$ that makes more sense. but now that after the 1st epoch the cost function was already minimized, how will the cost function further decrease during the 2nd epoch? $\endgroup$
    – feynman
    Mar 8, 2019 at 14:14
  • $\begingroup$ That is the point. Why are the model minimizing a loss function? Because we update our weights in a direction where we get more points 'right' and less wrong. And how is the intensity we update these weights? Through the partial derivatives of our loss function multiplied by A LEARNING RATE, therefore we update our loss function/weights through small steps, and not all at once. This is becausa in the second epoch we still have leverage to further decrease our loss. $\endgroup$
    – Victor Oliveira
    Mar 8, 2019 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.