What value does actually LSTM forget in a training phase? for example, I do have a surface temperature data for 10 years. Then I made them as a training data for building my neural network using lSTM algorithm. Then what value or which time step in my data that LSTM forget?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The way LSTMs are capable of learning long term dependencies is by keeping a cell state which serves as a memory of sorts.
This cell state is updated based on the values of different gates within the cell, forget gate being one of them.
The forget gate looks at the previous hidden state and the current input and outputs a number between 0 and 1 for each value in the cell state - deciding which values should be discarded and which saved.
During training, the cell learns which values in the cell state it should keep at different time steps in order to yield better results for the task at hand.
Now, saying exactly what are those values are is hard since it changes between samples and time steps.
For example, let's assume that your objective is to predict the temperature for tomorrow based on the previous 10 days. If you had an outlier on the 3rd day of the previous 10 days, that might hinder the prediction, and the cell could decide to forget that value. In other cases, it might decide to keep the value of the 3rd day since it helps prediction.
If you want to understand LSTMs better I suggest this great post by Chris Olah.
