I'm trying to understand the connection between loss function and backpropagation. From what I understood until now, backpropagation is used to get and update matrices and bias used in forward propagation in the LSTM algorithm to get current cell and hidden states. And loss function takes the predicted output and real output from the training set. But which part is the training part of the LSTM? Are they somehow connected ? What is the training objective for the LSTM model?
For sake of convenience, I'll go in ascending order of how the neural network is working
- The forward propagation of the LSTM cell is where the input is fed -> passed through the hidden states -> output is achieved (either at each time-step or at the end depending upon the type of problem like classification, forecasting, etc.,)
- Then after receiving the output the error between the actual value and the predicted value is calculated. This is where the loss function comes into the picture
- Classification problem - cross-entropy/log-likelihood
- Regression Problem - Mean Squared Error, Mean Absolute Error functions are used.
- So, after calculating the error only we backpropagate through time (BPTT) in case of RNN neural networks which updates weights of the matrices and cell states of the LSTM cells. In practice, we refer to these backpropagation algorithms as optimization algorithms like Gradient Descent, etc.,
- The optimization algorithms like RMSProp, Adam are much faster in practice than the standard gradient descent algorithm.
Overall, the training part of any neural network algorithm is that at the early stages of training, it will perform very horribly and over a period of time, with the correct loss function and backpropagation algorithm it will try to reduce the chances of classifying error and will converge fast depending upon the backpropagation algorithm used.