When trying to overfit the network, what is the practical maximum depth of an LSTM Neural network before it will start to fall apart? If possible, what is the state-of-the-art depth in LSTMs produced by companies like Google?
I know that in real world number of layers is implementation-dependent & number of neurons can be estimated, [1], [2] but curious if my results are typical:
I've implemented a system in C++, using iRProp+ (here is the paper, section 3.1), however, I am struggling to get the error-propagation under control when going above 10 layers.
I am just trying to overfit the network, it successfully and very quickly converges with 4-6 layers (~100 iRProp iterations and the error is down to 0.00001, where it started at around 4.00000)
The network tries to predict the next character in the alphabet (made from 26 characters), so each layer has an LSTM that works with 27-dimensional vectors. The error-propagation happens after 25 timesteps
If I crank-up to 10-14 layers, the error is really hesitant to even begin climbing down, and seems to simply oscillate around the 3.6 value. In incredibly rare cases if a weight initialization was lucky, with 10-14 layers the error will decrease, but usually it will just oscillate. Is that usual? I am using float
datatype, however, tested the double datatype and the oscilations still happen, so I doubt it's anything to do with precision. Adjusting the $n$ value (acceleration) doesn't seem to affect it either
Been looking at examples [3], but so far getting impression people use a maximum of 2-4 layers...