How to determine the inputshape of a ANN in Keras

Question

I started to use Keras for ANN and something that I do not really understand is which values to choose for the input_shape parameter of the first layer in an ANN? I know that the number should be equal to the inputs but how can I determine the order and the other value in the vector. For example, here is the code of a Multilayer Perceptron that takes 3 inputs and calculates 1 output

model.add(keras.layers.Flatten(input_shape=[3,])),
model.add(Dense(20,  activation='relu'))
model.add(Dense(40,  activation='relu'))
model.add(Dense(20,  activation='relu'))
model.add(Dense(3, activation='linear'))

In this case I use 'input_shape=[3,]'. The 3 comes from the 3 inputs but why do I have a ",]" as the second argument. Here you see on the other hand the code for a recurrent neural network that is used for time series forecasting and that has 6 input features for every timeslot of the time series and calculates 24 outputs:

model6 = keras.models.Sequential([
    keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 6]),
    keras.layers.SimpleRNN(20, return_sequences=True),
    keras.layers.TimeDistributed(keras.layers.Dense(24))
])

Why do I have here another order with "input_shape=[None, 6]" and not "input_shape=[6, None]" and why do I need here "None" as the first argument and not like in the multilayer perceptron just ",]". Has this something to do with the recurrent neural network such that I always have to use the number of input features as the second argument and 'None' as the first argument.

I read about this in the keras documentation but it is still confusing for me. Can you tell me how to choose those input_shape argument`I'd appreciate every comment.

Answer 1

There are basically two parts.

1. Why there is a comma in the [3,]? In this case you can skip it and just use [3]. You can encounter it in the tutorials, because you can pass the tuple as shape as well - (3,) and if you skip the comma in the tuple, then it will be just number, not tuple. So, it's just more of a python, not a keras itself. Try this in terminal

>>> 3 == (3)
True
>>> 3 == (3,)
False
>>> [3] == [3,]
True

The bottom line is that keras in this case expects 1-dimensional objects of size 3.

2. About the recurrent networks. In contrast with fully-connected neural nets, the input to the RNN is two-dimensional. I.e. you have a time series with 10 steps, each defined by 3 numbers, then the shape would be (10, 3). The problem is that you don't know beforehand how many steps you will have - it could be 20, it could be 5. So, you use None as a placeholder to say the keras to expect some dimension there.

You can ask, why we use None here and not something else. The reason is numpy has been using similar notations for years. See the example below

>>> import numpy as np
>>> x = np.ones(5)
>>> x.shape
(5,)
>>> x_2 = x[None, :5]
>>> x_2.shape
(1, 5)

In this case None in index tells numpy to add a dimension. It's not very transparent thing and could be confusing at first, especially with all the other notation, but you will get used to it.

EDIT

3. Worth mentioning, the number of dimensions of input tensors always will be bigger on one, i.e. 2 dimensions for MLP and 3 for the RNN, because we process data in batches. Thus, the first dimension always will be the number of samples. Let's look on the examples.

• For MLP input tensor shape could be [128, 3], where the 128 is the number of samples and 3 is number of features. input_shape=(3,)
• For RNN input tensor shape could be [128, 30, 3], where the 128 is the number of sample, 30 is sequence length and 3 is number of features. input_shape=(None, 3)
• For convolutional NN the inputs will be images and shape like [128, 220, 220, 3], where the 128 is the number of images, 220x220 - size of the image and 3 is number of channels (colors). input_shape=(220, 220, 3)

The interesting fact - we asked to specify the input shape not because keras authors are pedants, but because the specific size of the network is depend on it and we need this info during initialization. As batch size doesn't influence the model size, it is omitted by convention. We would probably omitted the sequence length as well, if the RNN was the only architecture, but for consistency with other type of models it is as it is.