0
$\begingroup$

I am tracing through matrix multiplication in forward propagation. For some context, I am thinking about a input shape of (1, 2, 100). I am curious how (2, 100) is propagated forward in a neural network.

I noticed that for a $m\times n$ matrix, $m$ is preserved throughout. Reading column wise, this would be a collection of vectors in $m$ dimensions. However, I am curious what $n$ contributes to the output matrix (or the matrices in between hidden layers)?

$\endgroup$
1
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Commented Jun 20, 2023 at 13:30

1 Answer 1

0
$\begingroup$

I assume you are talking about having vectors as inputs to the DNN, and not matrices.

If so, $m$ represents your batch size so how many inputs you forward to your network: each vector has $n$ components. So you get an $m\times n$ input matrix.

The size $m$ must be preserved all along the layers of the network because is the batch size, so the number of inputs that you provided as inputs. Therefore, (dense) layers actually work on the $n$ dimension, shrinking or expanding it.

Say your network has one hidden dense layer, its weight matrix would be $n\times u$, because it works on the dimensionality (num of components) of the vectors, transforming them from $n$ to $u$, resulting in an output (or hidden) matrix of shape $m\times u$.

$\endgroup$
2
  • $\begingroup$ I apologize, I am thinking of having matrices as inputs with a batch size of 1. I am curious on how a matrix is propagated forward assuming that it is multiplied by a weight matrix. For example, a shape of (1, 2, 100), how does (2, 100) matrix get propagated forward in the network. $\endgroup$
    – ajsdkh
    Commented Jun 17, 2023 at 14:51
  • $\begingroup$ @ajsdkh Right. Say you put a dense layer on top with $u$ units, so according to the "note" section here, the $(100, u)$ matrix is used as a kernel yielding $1\star 2$ sub-tensors, which are then reshaped to $(1,2,u)$. See also here for more details. $\endgroup$ Commented Jun 17, 2023 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.