1
$\begingroup$

I am new to deep learning & keras.

Refer to below code.

  1. I don't understand why yhat differs when I define the 1st layer input shape as 'input_shape' vs 'input_dim'. yhat should only be (1,1) - a single value.

  2. If instead, I use a simple RNN layer as my 1st layer, what should the inputs shape be?

Thank you

# univariate one step problem with mlp
from numpy import array
from keras.models import Sequential
from keras.layers import Dense
from keras.preprocessing.sequence import TimeseriesGenerator

# define dataset
series = array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) # 10 samples before processing by the Generator

# define generator
timestep = 2
generator = TimeseriesGenerator(series, series, length=timestep, batch_size=8)


# number of batch
print('Batches: %d' % len(generator))
# OUT --> Batches: 1

# print each batch
for i in range(len(generator)):
   x, y = generator[i]
   print('%s => %s' % (x, y))

#OUT:
[[1 2]
 [2 3]
 [3 4]
 [4 5]
 [5 6]
 [6 7]
 [7 8]
 [8 9]] => [ 3  4  5  6  7  8  9 10]
   
#After processing by the Generator, there are 8 samples in 1 batch. 

x, y = generator[0]
print(x.shape)
    
# define model
model = Sequential()

#TensorFlow assumes the first dimension is the batch_size which can have any size so you don't need to define it. The 2nd D is the number of time steps. The 3rd D is the number of features

#1st LAYER with input shape defined by input_shape
#model.add(Dense(100, activation='relu', input_shape= (timestep,1)))

#1st LAYER with input shape defined by input_dim
model.add(Dense(100, activation='relu', input_dim=timestep))

model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit_generator(generator, steps_per_epoch=1, epochs=200, verbose=0)
# make a one step prediction out of sample
x_input = array([9, 10]).reshape((1, timestep))
print(x_input.shape)

yhat = model.predict(x_input, verbose=0)
print(yhat)

# OUT: [[9.3066435, 10.239568]] if 1st layer's shape is input_shape
# OUT: [[11.545249]] if 1st layer's shape is input_dim
```
$\endgroup$

1 Answer 1

1
$\begingroup$

To clear this, we need to understand the difference between input_dim and input_shape, both are useful for two main reasons:

  1. Ensuring proper shapes are always passed.
  2. Initializing the weights without passing any dummy tensors(can call model.summary() after defining the architecture).
  • input_shape is used to tell the model what tensor shape should it expect.

  • input_dim is used to tell the model the number of dimensions of features.

More info about it is present here.

Coming back to your question, you'd see what the source of error is if you check model.summary() for both cases.

Here's your code modified for brevity:

  1. Creating dataset:
from numpy import array
from keras.models import Sequential
from keras.layers import Dense
from keras.preprocessing.sequence import TimeseriesGenerator

series = array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
timestep = 2
generator = TimeseriesGenerator(series, series, length=timestep, batch_size=8)
  1. Using input_shape to define your architecture:
model = Sequential()
model.add(Dense(100, activation='relu', input_shape=(timestep, 1)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

model.summary()

Output:

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 2, 100)            200       
                                                                 
 dense_1 (Dense)             (None, 2, 1)              101       
                                                                 
=================================================================
Total params: 301
Trainable params: 301
Non-trainable params: 0
_________________________________________________________________
  1. Using input_dim to define your architecture:
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=timestep))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

model.summary()

Output:

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_2 (Dense)             (None, 100)               300       
                                                                 
 dense_3 (Dense)             (None, 1)                 101       
                                                                 
=================================================================
Total params: 401
Trainable params: 401
Non-trainable params: 0
_________________________________________________________________

The problem is with you adding another dimension to your expected shape in input_shape. To correct it, you can pass a single-valued tensor as input_shape=(timestep,):

model = Sequential()
model.add(Dense(100, activation='relu', input_shape=(timestep,)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

model.summary()

Output:

Model: "sequential_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_4 (Dense)             (None, 100)               300       
                                                                 
 dense_5 (Dense)             (None, 1)                 101       
                                                                 
=================================================================
Total params: 401
Trainable params: 401
Non-trainable params: 0
_________________________________________________________________

You can cross-verify the results by fitting and checking with your own tensor as you did above.

The same logic applies to RNN, here's the input shape that it expects.

$\endgroup$
2
  • $\begingroup$ Can I use "input_dim" in a Dense layer with 3D input data? It doesn't seem to work when I tried. For 3D input data in a Dense layer, I can only use input_shape? Thanks! $\endgroup$
    – Peter
    Jan 9, 2022 at 6:56
  • $\begingroup$ input_dim expects integer or None, there might be a workaround but I think the simpler option is to directly use input_shape $\endgroup$ Jan 9, 2022 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.