I'm studying the Hochreiter-Schmidhuber long-short term memory recurrent architecture. The overall idea, information flow and manipulation is clear, and it seemingly works, but what I cannot understand that how could the effect of the respective "gates" are ensured, or indeed, checked one by one?

For example, it's clear that the "forget gate" (the first in the sequence of interactions between cell state, hidden state and new incoming training instance) is responsible for "forgetting" information from cell state, but I cannot see how it is the one that is actually forced to do the "forgetting", and not some other gate? The overall mechanism (unite x_t and h_t-1, squeeze this through some function and inject the result to c_t) is pretty similar in case of the input and output gates as well, so in theory (and in practice) maybe not the forget gate that does the forgetting, but perhaps the input gate in the specific case. How to explain and check this? Maybe catching the cell state immediately after the forget gate, compare it with c_t-1, and see if the vector values are somewhat decreased/nulled is an evidence? Or maybe the names of the gates are not necessary to be taken seriously but just interpret the whole mechanism as a random black box what somehow still works?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.