Is there any example of how Keras Dense layer handles 3D input.

The documentation explains the following:

If the input to the layer has a rank greater than 2, then Dense computes the dot product between the inputs and the kernel along the last axis of the inputs and axis 1 of the kernel (using tf.tensordot).

But I could not understand the internal matrix calculation

For example:

import tensorflow as tf
from tensorflow.keras.layers import Dense
sample_3d_input = tf.constant(tf.random.normal(shape=(4,3,2)))
dense_layer  = Dense(5)
op = dense_layer(sample_3d_input)

based on the documentation for a 3D input of shape (m,d0,d1), the shape of Layer's weight_matrix (or) kernel will have the shape (d1,units) which is (2,5) in this case. But I dont understand how the op is calculated to have the shape (m,d0,units)


In a regular fully connected layer (Dense), the computation is done using the following Matrix operation : $$R = A*W + B$$ With all matrixes being vectors (if batch size = 1), exept $W$, which has size(inputsize, outputsize).

for a 3D input, the way TF computes the output is simply by applying this formula only to the last dimension, considering all other dimensions as similar to batch sizes.

So in your example with input of size (4,3,2), Tensorflow 'flattens' the vector into a (12, 2) matrix, computes the result as if it was a 2D input, this gives a (12, 5) vector, and then 'unflattens' the vector to give back its 3 dimensions (4,3,5).

  • $\begingroup$ Thanks for the answer. When I try to flatten the matrix in numpy its just produces a list. Can you please explain how to flatten and unflatten, or even any references on how to do that will be great $\endgroup$ Apr 27 at 0:29
  • $\begingroup$ I got it now. So when you say flatten its actually reshape right? $\endgroup$ Apr 27 at 0:44
  • $\begingroup$ Yep exactly, this is more of a reshape than a flatten $\endgroup$
    – Ubikuity
    Apr 27 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.