1
$\begingroup$

When trying to overfit the network, what is the practical maximum depth of an LSTM Neural network before it will start to fall apart? If possible, what is the state-of-the-art depth in LSTMs produced by companies like Google?

I know that in real world number of layers is implementation-dependent & number of neurons can be estimated, [1], [2] but curious if my results are typical:

I've implemented a system in C++, using iRProp+ (here is the paper, section 3.1), however, I am struggling to get the error-propagation under control when going above 10 layers.

I am just trying to overfit the network, it successfully and very quickly converges with 4-6 layers (~100 iRProp iterations and the error is down to 0.00001, where it started at around 4.00000)

The network tries to predict the next character in the alphabet (made from 26 characters), so each layer has an LSTM that works with 27-dimensional vectors. The error-propagation happens after 25 timesteps

If I crank-up to 10-14 layers, the error is really hesitant to even begin climbing down, and seems to simply oscillate around the 3.6 value. In incredibly rare cases if a weight initialization was lucky, with 10-14 layers the error will decrease, but usually it will just oscillate. Is that usual? I am using float datatype, however, tested the double datatype and the oscilations still happen, so I doubt it's anything to do with precision. Adjusting the $n$ value (acceleration) doesn't seem to affect it either

Been looking at examples [3], but so far getting impression people use a maximum of 2-4 layers...

$\endgroup$
4
  • $\begingroup$ Just curious, what in the world are you trying to do? Why do you want to overfit? Are you just trying to memorize input-output mappings? $\endgroup$
    – tom
    Dec 19, 2017 at 6:42
  • $\begingroup$ This is my first custom-built network, so I am basically debugging it. I have found quite a few errors already which originally seemed like a usual thing, but other people pointed out to me that it's not the expected behavior. A simple overfitting task is quite good for uncovering inconsistencies. So I am trying to get a feeling whether I have hit the limits of LSTM & have to read more papers on optimization, or there is an error somewhere in my code $\endgroup$
    – Kari
    Dec 19, 2017 at 7:22
  • $\begingroup$ Found an article (link) about Batch normalization, mentions 152 layers used by Microsoft (but I assume it's a feed-forward net) $\endgroup$
    – Kari
    Dec 26, 2017 at 10:33
  • $\begingroup$ I just found 1 issue - in my custom implementation, I carried forward the cell state, so the next forward prop doesn't begin with an empty cell. That's something as "stateful cell" in Keras framework. However, resetting it to zero helped indeed. Now I am able to stack any number of layers (with appropriate learning rate of course) $\endgroup$
    – Kari
    Jan 31, 2018 at 0:18

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.