My question is does LSTM by nature, also take previous output values besides feature vector as new feature element for the next output values?
You didn't specify which framework you are using, so I will explain for a custom (c++ etc) implementation.
At each timestep LSTM will:
- Take a new input from you (for example, as a one-hot encoded vector)
- Internally fetch a separate vector which is the LSTM's output from Timestep [t-1]
- These two vectors are like a 'fuel' and will be used to shape out a Cell at [t]
- The result gets produced for this Timestep [t]
Back Propagation creates gradients for the weights within the LSTM, but also gradients for these two vectors.
Because there are multiple timesteps, you get a Gradient during each Timestep, which has to be added (component-wise) before being used.