I wanted to know when to use dot product and when to not , I also don't know when we must transpose an array and why should we , could someone help me to understand this ? If you could give me the derivative of cost function with respect to each Weigh (W1, W2, W3) using notations bellow to compare it with the results I found I'll appreciate that !

neural network

  • 1
    $\begingroup$ To match the dimensions, we need to transform the arrays/weights accordingly $\endgroup$ – Aditya Aug 28 '18 at 2:42

Neuron-level operations:

First, let's describe the output of each neuron in the network individually.

For the first hidden layer:

$$ A_2^1 = f_2(Z_2^1) = f_2(X \cdot W_1^1)$$ $$ A_2^2 = f_2(Z_2^2) = f_2(X \cdot W_1^2)$$ $$ A_2^3 = f_2(Z_2^3) = f_2(X \cdot W_1^3)$$

Let me explain the notation a bit: $A_2^1$ is the output of the first neuron of the second layer. $f_2$ is the activation function of the second layer. $W_1^2$ is line connecting the second neuron of the weights you denote $W1$ in the figure.

Second hidden layer:

$$ A_3^1 = f_3(Z_3^1) = f_3(A_2^1 \cdot W_2^{11} + A_2^2 \cdot W_2^{21} + A_2^3 \cdot W_2^{31})$$ $$ A_3^2 = f_3(Z_3^2) = f_3(A_2^1 \cdot W_2^{12} + A_2^2 \cdot W_2^{22} + A_2^3 \cdot W_2^{32})$$ $$ A_3^3 = f_3(Z_3^3) = f_3(A_2^1 \cdot W_2^{13} + A_2^2 \cdot W_2^{23} + A_2^3 \cdot W_2^{33})$$

Again $A_3^1$ is the output of the first neuron of the third layer and $f_3$ is the activation function of the the third layer. This time $W_2^{12}$ corresponds to the weight $W2$ that connects the first neuron of the previous layer to the second neuron of the current layer.

For the output layer:

$$ \hat y = A_4 = f_4(Z_4) = f_4(A_3^1 \cdot W_3^1 + A_3^2 \cdot W_3^2 + A_3^3 \cdot W_3^3)$$

The notation follows the rules above.

Note that all the above operations are between simple numbers, meaning that all $\cdot$ symbols you saw above are common multiplications! Because this way of representing Neural Networks isn't effective, instead of describing each neuron individually, we usually describe the operations of layers with vectors or arrays. This is where your question lies (about the dot product and the transposing).

Layer-level operations:

First hidden layer:

$$ A_2 = f_2(Z_2) = f_2(X \cdot W_1) $$

In the above equation $W_1$, $Z_2$ and $A_2$ are all $1 \times 3$ arrays. While $X$ remains a simple number (you can think of it as a $1 \times 1$ array).

Second hidden layer:

$$ A_3 = f_3(Z_3) = f_3(A_2 \cdot W_2) $$

Again $A_3$, $Z_3$ and $A_2$ are $1 \times 3$ arrays. This time $W_2$ is a $3 \times 3$ array.

Output layer:

$$ A_4 = f_4(Z_4) = f_4(A_3 \cdot W_3) $$

Here, $A_4$ and $A_4$ are numbers and $W_3$ is a $3 \times 1$ array.

Every $\cdot$ in the above is a basic matrix multiplication. Because this example wasn't the best to answer your question, I'm going to change it a bit. Imagine if the second hidden layer had $4$ neurons instead of $3$. In order for us to get from a $1 \times 3$ array to a $1 \times 4$ one we would need a weight array with a shape of $3 \times 4$.

$$ \begin{pmatrix} A_2^1 & A_2^2 & A_2^3 \end{pmatrix} \cdot \begin{pmatrix} W_2^{11} & W_2^{12} & W_2^{13} & W_2^{14}\\ W_2^{21} & W_2^{22} & W_2^{23} & W_2^{24}\\ W_2^{31} & W_2^{32} & W_2^{33} & W_2^{34} \end{pmatrix} = \begin{pmatrix} Z_3^1 & Z_3^2 & Z_3^3 & Z_3^4 \end{pmatrix} $$

This is summarized as $A_2 \cdot W_2 = Z_3$. Because in matrix multiplication the order of the two matrices counts, we need to make sure that the operation is possible. If they are in a reverse order we need to transpose the weight array. This would result in the following equation: $W_2^T \cdot A_2 = Z_3$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.