Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
You are updating your network parameters, that is, weights for fully connected layers, for the filters in the convolution operations, etc.
The hyperparameters are fixed once you start training your network. Hyperparameters are not intrinsic to the learning process and is something that the practitioner should tune carefully with GridSearch, Bayesian Optimization and Cross-Validation techniques.
You have just one loss function during training, and at each batch procesing you update your weights correcting your network and, at least theoretically, diminishing your loss function. So after the first epoch, you have reached a certain value, that will be update on the next epoch.
Think as you are on the top of a mountain, and you are climbing down, to no get tired, you count 10 steps and rest a little, after 10 steps you are not on the top again, you are going down from where you stopped, right? That is an analogy (I think it is bad, but if you understand it is ok haha).