It is known that RNNs do not have long memory, that is they do not capture long dependencies.
Usually this is explained by the vanishing (or explolding) gradients problem - when computing the gradients of the RNN, the early part of the input signal doesn't contribute to the gradient because of many multipications in the back propagation process.
However, I think that it may be only a simptom of the real problem. The gradients are small because the early part of the signal really does not affect the output (because of forward multipications). This is not only a point of view difference, it is an essential difference - this is not an optimization problem but a modeling problem. I think that even if we had a "magic initializer" that would initialize our RNN with the best possible weights it would still suffer from short memory problems, hence the gradients are not the major problem.
Is the RNN short memory problem really an optimization problem or a modeling problem?
To make it clearer - a similar question - Do exist weights such that RNN with those weights has long memory?
