# Number of parameters keras dense layer with a 2D input

I am using 2D data in a classification problem using keras.

So I am defining a keras model as following:

in_ = Input((5, 10))
out = Dense(100, activation='relu', name = 'dense_1')(in_)

model = Model(in_, out)
model.summary()


which returns a compiled model with the following parameters:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_9 (InputLayer)         (None, 5, 10)             0
_________________________________________________________________
dense_1 (Dense)              (None, 5, 100)            1100
=================================================================
Total params: 1,100
Trainable params: 1,100
Non-trainable params: 0
_________________________________________________________________


What I don't understand is why the dense_1 layer has only 1100 parameters and not 5100 parameters. What I was expecting is that the Dense Layer is going to connect to all the inputs 50 (5*10=50 inputs) giving a number of parameters of 5100 (100*50+100=5100, weights + biases). So apparently the Dense Layer only connects to the last dimension of the input? What happens in the other dimension?

If I flatten the input layer I obtain my expected number of parameters:

in_ = Input((5,10))
x = Flatten()(in_)
out = Dense(100, activation='relu', name = 'dense_1')(x)

model = Model(in_, out)
model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_13 (InputLayer)        (None, 5, 10)             0
_________________________________________________________________
flatten_6 (Flatten)          (None, 50)                0
_________________________________________________________________
dense_1 (Dense)              (None, 100)               5100
=================================================================
Total params: 5,100
Trainable params: 5,100
Non-trainable params: 0
_________________________________________________________________


So what is going on with a Dense Layer when the previous layer has more than one dimension? What happens with the dimensions and the dot products and biases? Why does the number of parameters changes?

Yes, it takes only to the last dimension, accordingly to the source code (comments are mine): https://github.com/keras-team/keras/blob/88af7d0c97497b5c3a198ee9416b2accfbc72c36/keras/layers/core.py#L880

def build(self, input_shape):
assert len(input_shape) >= 2
input_dim = input_shape[-1]  # uses last dimension

• But in that case how the dot product is performed? I mean, how to you perform the dot product when you have a 2D matrix? when is a 1D array is easy because is $$\vec{x}\dot\vec{w}$$ but when $x$ is 2D which dimension do you choose? – Iván Dec 13 '18 at 23:17
• There is no problem having a 2D matrix, it will be a dot product between matrices. If X have shape (a, b) and W have shape (b, c) then the result will be a matrix of shape (a, c). – xboard Dec 15 '18 at 0:33

The first dimension is expected to be the batch size. And as said in the documentation and by @xboard, only the last dimension contributes to the size of the weights.

# Input shape

nD tensor with shape: (batch_size, ..., input_dim).
The most common situation would be
a 2D input with shape (batch_size, input_dim).


# Output shape

nD tensor with shape: (batch_size, ..., units).
For instance, for a 2D input with shape (batch_size, input_dim),
the output would have shape (batch_size, units).


(Source)

Basically the input shape of X is 5 x 10 matrix, the output shape of Y is 5 x 100 so:

i) The weight W of 10 x 100 shape will yield 1000 parameters, then plus the 100 bias B (Y = W*X + B) ii) The inter-connection happens as the individual ij element of that W matrix multiplies with the input iii) Whether you say it interconnect at last dimension is just a matter of wording misunderstanding, as you can tell from the matrix multiplication rule all input get multiplied.

Note: Matrix multiplication rule (n x m) * (m x k) = (n x k) dimension