Number of parameters keras dense layer with a 2D input

Question

I am using 2D data in a classification problem using keras.

So I am defining a keras model as following:

in_ = Input((5, 10))
out = Dense(100, activation='relu', name = 'dense_1')(in_)

model = Model(in_, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()

which returns a compiled model with the following parameters:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_9 (InputLayer)         (None, 5, 10)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 5, 100)            1100      
=================================================================
Total params: 1,100
Trainable params: 1,100
Non-trainable params: 0
_________________________________________________________________

What I don't understand is why the dense_1 layer has only 1100 parameters and not 5100 parameters. What I was expecting is that the Dense Layer is going to connect to all the inputs 50 (5*10=50 inputs) giving a number of parameters of 5100 (100*50+100=5100, weights + biases). So apparently the Dense Layer only connects to the last dimension of the input? What happens in the other dimension?

If I flatten the input layer I obtain my expected number of parameters:

in_ = Input((5,10))
x = Flatten()(in_)
out = Dense(100, activation='relu', name = 'dense_1')(x)

model = Model(in_, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_13 (InputLayer)        (None, 5, 10)             0         
_________________________________________________________________
flatten_6 (Flatten)          (None, 50)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 100)               5100      
=================================================================
Total params: 5,100
Trainable params: 5,100
Non-trainable params: 0
_________________________________________________________________

So what is going on with a Dense Layer when the previous layer has more than one dimension? What happens with the dimensions and the dot products and biases? Why does the number of parameters changes?

Answer 1

Yes, it takes only to the last dimension, accordingly to the source code (comments are mine): https://github.com/keras-team/keras/blob/88af7d0c97497b5c3a198ee9416b2accfbc72c36/keras/layers/core.py#L880

def build(self, input_shape):
    assert len(input_shape) >= 2
    input_dim = input_shape[-1]  # uses last dimension

Answer 2

The first dimension is expected to be the batch size. And as said in the documentation and by @xboard, only the last dimension contributes to the size of the weights.

Input shape

nD tensor with shape: `(batch_size, ..., input_dim)`.
The most common situation would be
a 2D input with shape `(batch_size, input_dim)`.

Output shape

nD tensor with shape: `(batch_size, ..., units)`.
For instance, for a 2D input with shape `(batch_size, input_dim)`,
the output would have shape `(batch_size, units)`.

(Source)

Answer 3

Basically the input shape of X is 5 x 10 matrix, the output shape of Y is 5 x 100 so:

i) The weight W of 10 x 100 shape will yield 1000 parameters, then plus the 100 bias B (Y = W*X + B) ii) The inter-connection happens as the individual ij element of that W matrix multiplies with the input iii) Whether you say it interconnect at last dimension is just a matter of wording misunderstanding, as you can tell from the matrix multiplication rule all input get multiplied.

Note: Matrix multiplication rule (n x m) * (m x k) = (n x k) dimension