What are the differences between Convolutional1D, Convolutional2D, and Convolutional3D?

Question

I've been learning about Convolutional Neural Networks. When looking at Keras examples, I came across three different convolution methods. Namely, 1D, 2D & 3D. What are the differences between these three layers? What are their use cases? Are there some links or references to show their use cases?

Answer 1

The only difference is the dimensionality of the input space. The input for a convolutional layer has the following shape:

input_shape = (batch_size,input_dims,channels)

• Input shape for conv1D: (batch_size,W,channels)

  Example: 1 second stereo voice signal sampled at 44100 Hz, shape: (batch_size,44100,2)

• Input shape for conv2D: (batch_size,(H,W),channels)

  Example: 32x32 RGB image, shape: (batch_size,32,32,3)

• Input shape for conv3D: (batch_size,(H,w,D),channels)

  Example (more tricky): 1 second video of 32x32 RGB images at 24 fps, shape: (batch_size,32,32,3,24)

What a channel is?

The key thing is to think about what the channel means for our input data. The convolutional layer apply different filters for each channel, thus, the weights of the conv layer have the following shape:

(kernel_size,num_channels,num_filter_per_channels)

Example:

Convolutional layer with 12 filters and square kernel matrix of size of 3. This layer will apply 12 different filters for each channel. In the examples given previously:

• 1 second stereo voice signal sampled at 44100 Hz, kernel_size = 3

  12 x 2 = 24 one-dimensional filters, 12 filter for each channel

   Weigths shape: (3, 2, 12)
  

• 32x32 RGB image, kernel_size = (3,3)

  12 x 3 = 36 two-dimensional filters, 12 filter for each channel

  Weights shape: (3, 3, 3, 12)
  

• 1 second video of 32x32 RGB images at 24 fps, kernel_size = (3,3,3)

  24 x 12 = 288 three-dimensional filters, 12 filter for each channel

  Weights shape: (3, 3, 3, 24, 12)
  

Thus deciding what a channel means is very important, since each channel has its own set of filters. For the first examples, it seems straightforward to decide that the stereo signals and the RGB images are different channels... they are commonly named like that (stereo channels, RGB channels) indeed. In the video example, it is more ambiguous... Setting a video as a 3D input with the temporal dimension as channel may not be the best option since in that way, the order in which temporal frames come does not matter (the outputs for the filters of each channel are summed up) resulting in losing the intrinsic temporal dynamics of the input data . One better approach (depending on the application) is to process the RGB images with 2D convolutions in a recurrent neural network. The same happens with the voice signal, which rarely is processed in a neural network with Conv1D layers, in favor of recurrent approaches.

It is important to note that a signal with an input dimension D can be regarded as a signal of D+1 dimension with one channel, but the resulting feature space may be less representative/useful:

 (44100,2) --> expand_dimension(axis=-1)--> ((44100,2),1)

Keras code suporting the examples

from keras import Input, Conv1D, Conv2D, Conv3D
#1D
in_ = Input(shape=(44100,2))
layer = Conv1D(filters=12,kernel_size=3)
out_ = layer(in_)
print("Weights shape: {}".format(layer.get_weights()[0].shape))

#2D
in_ = Input(shape=(32,32,3))
layer = Conv2D(filters=12,kernel_size=3)
out_ = layer(in_)
print("Weights shape: {}".format(layer.get_weights()[0].shape))

#3D
in_ = Input(shape=(32,32,3,24))
layer = Conv3D(filters=12,kernel_size=3)
out_ = layer(in_)
print("Weights shape: {}".format(layer.get_weights()[0].shape))

Answer 2

Conv1D is used for input signals which are similar to the voice. By employing them you can find patterns across the signal. For instance, you have a voice signal and you have a convolutional layer. Each convolution traverses the voice to find meaningful patterns by employing a cost function.

Conv2D is used for images. This use case is very popular. The convolution method used for this layer is so called convolution over volume. This means you have a two-dimensional image which contains multiple channels, RGB as an example. In this case, each convolutional filter should be a three-dimensional filter to be convolved, cross-correlated actually, with the image to find appropriate patterns across the image.

Conv3D is usually used for videos where you have a frame for each time span. These layers usually have more parameters to be learnt than the previous layers. The reason we call them $3D$ is that other than images for each frame, there is another axis called time containing discrete values, and each of them corresponds to a particular frame.