Why do RNNs usually have fewer hidden layers than CNNs?

Question

CNNs can have hundreds of hidden layers and since they are often used with image data, having many layers captures more complexity.

However, as far as I have seen, RNNs usually have few layers e.g. 2-4. For example, for electrocardiogram (ECG) classification, I've seen papers use LSTMs with 4 layers and CNNs with 10-15 layers with similar results.

Is this because RNNs/LSTMs are harder to train if they are deeper (due to gradient vanishing problems) or because RNNs/LSTMs tend to overfit sequential data fast?

Answer 1

CNNs and RNNs feature extraction methods:

CNNs tend to extract spatial features. Suppose, we have a total of 10 convolution layers stacked on top of each other. The kernel of the 1st layer will extract features from the input. This feature map is then used as an input for the next convolution layer which then again produces a feature map from its input feature map.

Likewise, features are extracted level-by-level from the input image. If the input is a small image of 32 * 32 pixels, then we will definitely require fewer convolution layers. A bigger image of 256 * 256 will have comparatively higher complexity of features.

RNNs are temporal feature extractors as they hold a memory of the past layer activations. They extract features like an NN, but RNNs remember the extracted features across timesteps. RNNs could also remember features extracted via convolution layers. Since they hold a kind-of memory, they persist in temporally/time features.

---

In case of electrocardiogram classification:

On the basis of the papers you read, it seems that,

ECG data could be easily classified using temporal features with the help of RNNs. Temporal features are helping the model to classify the ECGs correctly. Hence, the usage of RNNs is less complex.

The CNNs are more complex because,

The feature extraction methods used by CNNs lead to such features which are not powerful enough to uniquely recognize ECGs. Hence, the larger number of convolution layers is required to extract those minor features for better classification.

At last,

A strong feature provides less complexity to the model whereas a weaker feature needs to be extracted with complex layers.

---

Is this because RNNs/LSTMs are harder to train if they are deeper (due to gradient vanishing problems) or because RNNs/LSTMs tend to overfit sequential data fast?

This could be taken as a thinking perspective. LSTM/RNNs are prone to overfitting in which one of the reasons could be vanishing gradient problem as mentioned by @Ismael EL ATIFI in the comments.

I thank @Ismael EL ATIFI for the corrections.

Answer 2

About the number of layers

The reason can be understood by looking at the architecture of a CNN and an LSTM and how the might operated on time-series data. But I should say that the number of layers is something that depends heavily on the problem you are trying to solve. You might be able to solve an ECG classification using few LSTM layers, but for activity recognition from videos you will need more layers.

Putting that aside, here's how a CNN and an LSTM might process a time series signal. A very simple signal where after three positive cycles you get a negative cycle.

For a CNN to see this simple pattern it takes 4 layers in this example. When a CNN processes a time series input, a convolution outputs does not know about the previous outputs (i.e. they are not connected). However an LSTM can do that just using a single layer as they can remember temporal patterns up to 100s of time steps. Because one output is based on the current input as well as the previous inputs the model has seen.

I am not saying this is the only reason, but it is probably one of the main factors why CNNs require more layers and LSTMs don't for time series data.

About vanishing gradients and overfitting

Vanishing gradient is likely to become a problem within a single layer than across layers. That is when processing many sequential steps the knowledge about the first few steps will likely to disappear. And I don't think sequential models are likely to overfit on time-series data if you regularize them correctly. So this choice is probably more influenced by the architecture/capabilities of the models than by the vanishing gradient or overfitting.

Answer 3

I see 2 possible reasons for why a RNN could necessitate fewer layers than a CNN to reach the same performance:

1. RNN layers are generally fully connected layers which have more parameters than a convolutional layer.

2. A RNN has some components which allow to combine inputs by multiplication unlike a CNN which can only combine inputs by weighted addition. Thus the multiplicative capacity of the RNN gives him more "power" to compute features. The CNN would need many layers to "imitate" this multiplicative capacity.