I am performing gradient check for my LSTM which has 4 timesteps. The LSTM looks as follows:
01 01 01 01
^ ^ ^ ^
LSTM --> LSTM --> LSTM --> LSTM
^ ^ ^ ^
11 11 11 11
So, at every timestep we are feeding in vector {1,1} and expect {0,1} at the output.
Assume I perturb the weight inside LSTM, then perform 4 forward props - one for each timestep - how do I now get delta of the cost function that this single perturbation has caused?
Am I allowed to simply add-up the change in Cost from all 4 timesteps to treat it as derivative estimate?
Also, should I perform it as follows for LSTM:
- perturb a single weight upwards
- forward prop 4 timesteps
- perturb the weight downwards
- forward prop 4 timesteps
- get 4 deltas
- sum the 4 deltas to get a total change in Cost
or
- Set N=0
- perturb the weight upwards
- foward prop at a particular timestep N
- perturb the weight downwards
- forward prop at a particular timestep N
- get single delta, store it away
- increment N
- until N not equal 4 return to step 2)
- sum the 4 deltas to get a total change in Cost
The second approach somehow seems more correct, because LSTM will have a hidden state ..Is this correct intuition or it won't matter?