Looking at the literature, there are 2 distinct approaches to LSTM
Having tried the first approach (I still used standard weights on dataGate but no recurrent weights in that place) seems fine but a little odd - network converges well, but very rarely jumps over local mimima, as if too high learning rate.
Personally I like the second approach (recurrent weights on all 4 entry points), which one do I use?
to make things worse, the "LSTM Peephole paper", page 121 describes 3 connections (no peephole for dataGate).
This destroys the equal-sizes of my matrices, because now 3 gates use the Usual, Recurrent and Peephole weights, but the very first gate only uses the Usual weights