building a 2-layer LSTM for time series prediction using tensorflow

Question

From Tensorflow tutorials i am experimenting time series with LSTM

In the section 'multi-step prediction' using LSTM tutorial says

Since the task here is a bit more complicated than the previous task, the model now consists of two LSTM layers. Finally, since 72 predictions are made, the dense layer outputs 72 predictions.

where previous task was prediction over a single point.

How do we know how many layers a problem requires (here, 2) ?

Then, from implementation point of view, using Python Tensorflow library,

multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(32,
                                          return_sequences=True,
                                          input_shape=x_train_multi.shape[-2:]))
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
multi_step_model.add(tf.keras.layers.Dense(72))

why is there a need for the adding a Dense(72) layer ? what is the function Dense() doing ? (reading the doc doesn't help really)

Answer 1

First question: How many layers?

This is architectural question and one of them most important when constructing NN. Generally the more complex the task the more layers you should use to approximate (until a certain point than there is overkill, motivation for ResNet) If you are looking for some guidelines there are some good posts, but the research and general trend nowadays is that we are over-doing it in the first place, and we can achieve some good results with some smart tricks without making it too deep.

TL;DR it depends on the problem, but not as deep as we think for 98% of problems.

Second (third?) question: why is there a need for the adding a Dense(72) layer ? what is the function Dense() doing ? Well you said that Finally, since 72 predictions are made, the dense layer outputs 72 predictions. For the question what is dense function doing in short its producing an (output) vector. In long dense layer represents a matrix vector multiplication. Values in the matrix are actually the trainable parameters (weights) which get updated during backpropagation, and if you seen mathematically representation of NN with matrizes (which all of them are thats how you utilise power of GPU-s bla bla) thats exactly what this dense layer represents.