If an epoch is defined as the neural network training process after seeing the whole training data once. How is it that when starting the next epoch, the loss is almost always smaller than the first one? Does this mean that after an epoch the weights of the neural network are not reset? and each epoch is not a standalone training process?
$\begingroup$
$\endgroup$
asked May 1, 2020 at 13:01
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
1
An epoch is not a standalone training process, so no, the weights are not reset after an epoch is complete. Epochs are merely used to keep track of how much data has been used to train the network. It's a way to represent how much "work" has been done.
Epochs are used to compare how "long" it would take to train a certain network regardless of hardware. Indeed, if a network takes 3 epochs to converge, it will take 3 epochs to converge, regardless of hardware. If you had used time, it would be less meaningful as one machine could maybe do 1 epoch in 10 minutes, and another setup might only do 1 epoch in 45 minutes.
Neural networks (sadly) are usually not able to learn enough by seeing the data once, which is why multiple epochs are often required. Think about it as if you were studying a syllabus for a course. Once you finished the syllabus (first epoch), you go over it again to understand it even better (epoch 2, epoch 3, etc.)
answered May 1, 2020 at 13:20
-
$\begingroup$ I never said epochs = time, will edit to make it clearer. I definitely agree that epoch is a unit of work. I was trying to say that using epoch is better than using time because time is less comparable across different hardware $\endgroup$ May 1, 2020 at 17:06
$\begingroup$
$\endgroup$
6
How is it that when starting the next epoch, the loss is almost always smaller than the first one? Does this mean that after an epoch the weights of the neural network are not reset?
Yes. The network weights are initialized once before the training starts. After every iteration, the weights are updated by backpropagation using the error gradients that you obtain from the batch of data fed to the network at that iteration. Once an epoch is done, the weights are now better optimized to your training data, meaning you get a lower training loss. The next epoch builds on the weights you got after the first epoch to improve the performance further. This is why the loss the will keep decreasing as the network is trained for more epochs (assuming the hyperparameters are set properly).
each epoch is not a standalone training process?
Yes. An epoch is a part of the training process. You improve the network's performance by training it for as many epochs as necessary to achieve the desired performance.
-
$\begingroup$ Maybe I'm being pedantic, but I wouldn't say an epoch is a standalone training process. To me, an epoch is simply a unit of time that is comparable as it isn't dependent on hardware. Epochs can actually be sliding windows, let's say that you have 10 data points and you don't shuffle them for the sake of example. You'll have an epoch between 0 and 10, but also between 1 and 11, and so on. $\endgroup$ May 1, 2020 at 13:22
-
$\begingroup$ Have edited my answer :) With regards to hardware dependency, since batch size and therefore an epoch depends on the GPU memory, doesn't that mean that hardware comes into play whether an epoch runs or not? $\endgroup$ May 1, 2020 at 13:32
-
$\begingroup$ An epoch is neither dependent on your GPU nor your batch size. The batch size just changes how many iterations will be needed to complete an epoch. If I have 100 data points as my train set, an epoch will be completed whenever I trained on these 100 data points. Whether I do it one data point at a time, or in one go doesn't affect the epoch. $\endgroup$ May 1, 2020 at 14:37
-
$\begingroup$ If you choose a batch size that's too big for the GPU to handle, you run into an OOM error. My line of thinking is that if this happens, then you cant do an epoch of training because the batch size is too big for the GPU you have. But if you choose a smaller batch size, then an epoch can be completed. Choosing an appropriate batch size makes the difference between an epoch being completed or not. This is a roundabout way of saying that completing an epoch is indirectly dependent on hardware. Now I feel maybe I am being pedantic... $\endgroup$ May 1, 2020 at 14:49
-
1$\begingroup$ Makes sense. I concede :) Thanks for the clarification :D $\endgroup$ May 1, 2020 at 14:58
$\begingroup$
$\endgroup$
Yes, they don’t reset it. They train on the same set of weights continuously.
An epoch means the model completed training on the entire dataset once.
The loss is smaller because the model improves.
$\begingroup$
$\endgroup$
The concept of epoch is related to the cycle who wants all your training and testing base lots. You only need to reset the network parameters when your database is changed, for example in cross-validation. In the CV for each k iterations the parameters need to be reset.
