How is the input gate neural network trained what to remember by propagating the error rate from predicting the next word in the language model? How does it help it to learn if it remembered the right data or not?
I imagine maybe doing it heuristically. For example, when the full sentence is -
"It takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"
And the input to the language model is -
"It takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must "
But it outputs "go" for the next word instead of "run" I can see that it forgot and didn't save the "running" word earlier. So in this scenario, I can take the stem of the next word which is "run" and search it backward in the sentence to see where it forgot (or didn't save) the "running" word, then I can penalize the NNs and train them like that.
What do you think? What am I missing here?