2
$\begingroup$

I am using 2D data in a classification problem using keras.

So I am defining a keras model as following:

in_ = Input((5, 10))
out = Dense(100, activation='relu', name = 'dense_1')(in_)

model = Model(in_, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()

which returns a compiled model with the following parameters:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_9 (InputLayer)         (None, 5, 10)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 5, 100)            1100      
=================================================================
Total params: 1,100
Trainable params: 1,100
Non-trainable params: 0
_________________________________________________________________

What I don't understand is why the dense_1 layer has only 1100 parameters and not 5100 parameters. What I was expecting is that the Dense Layer is going to connect to all the inputs 50 (5*10=50 inputs) giving a number of parameters of 5100 (100*50+100=5100, weights + biases). So apparently the Dense Layer only connects to the last dimension of the input? What happens in the other dimension?

If I flatten the input layer I obtain my expected number of parameters:

in_ = Input((5,10))
x = Flatten()(in_)
out = Dense(100, activation='relu', name = 'dense_1')(x)

model = Model(in_, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_13 (InputLayer)        (None, 5, 10)             0         
_________________________________________________________________
flatten_6 (Flatten)          (None, 50)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 100)               5100      
=================================================================
Total params: 5,100
Trainable params: 5,100
Non-trainable params: 0
_________________________________________________________________

So what is going on with a Dense Layer when the previous layer has more than one dimension? What happens with the dimensions and the dot products and biases? Why does the number of parameters changes?

$\endgroup$
1
$\begingroup$

Yes, it takes only to the last dimension, accordingly to the source code (comments are mine): https://github.com/keras-team/keras/blob/88af7d0c97497b5c3a198ee9416b2accfbc72c36/keras/layers/core.py#L880

def build(self, input_shape):
    assert len(input_shape) >= 2
    input_dim = input_shape[-1]  # uses last dimension
$\endgroup$
  • $\begingroup$ But in that case how the dot product is performed? I mean, how to you perform the dot product when you have a 2D matrix? when is a 1D array is easy because is $$\vec{x}\dot\vec{w}$$ but when $x$ is 2D which dimension do you choose? $\endgroup$ – Iván Dec 13 '18 at 23:17
  • $\begingroup$ There is no problem having a 2D matrix, it will be a dot product between matrices. If X have shape (a, b) and W have shape (b, c) then the result will be a matrix of shape (a, c). $\endgroup$ – xboard Dec 15 '18 at 0:33
0
$\begingroup$

The first dimension is expected to be the batch size. And as said in the documentation and by @xboard, only the last dimension contributes to the size of the weights.

Input shape

nD tensor with shape: `(batch_size, ..., input_dim)`.
The most common situation would be
a 2D input with shape `(batch_size, input_dim)`.

Output shape

nD tensor with shape: `(batch_size, ..., units)`.
For instance, for a 2D input with shape `(batch_size, input_dim)`,
the output would have shape `(batch_size, units)`.

(Source)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.