Considering a simple ANN:
$$x \rightarrow f=(U_{m\times n}x^T)^T \rightarrow g = g(f) \rightarrow h = (V_{p \times m}g^T)^T \rightarrow L = L(h,y) $$
where $x\in\mathbb{R}^n$, $U$ and $V$ are matrices, $g$ is the point-wise sigmoid function, $L$ returns a real number indicating the loss by comparing the output $h$ with target $y$, and finally $\rightarrow$ represents data flow.
To minimize $L$ over $U$ and $V$ using gradient descent, we need to know $\frac{\partial L}{\partial U_{ij}}$ and $\frac{\partial L}{\partial V_{ij}}$, I know two ways to do this:
- do the differentiation point wise, and having a hard time figuring out how to vectorize it
- flatten $U$ and $V$ into a row vector, and use multivariate calculus (takes a vector, yields a vector) to do the differentiation
For the purpose of tutorial or illustration, the above two methods might be suffice, but say if you really want to implement back-prop by hand in the real world, what math will you use to do the derivative? I mean, is there a branch, or method in meth, that teaches you how to take derivative of vector-valued function of matrices?