this note says this

$$ \frac{\partial \textbf{z}}{\partial \textbf{x}} = \text{diag}(f'(\textbf{x})) $$

I know this means make a vector to square matrix which has an original vector as diagonal component.

but, since $\textbf{x}$ is a vector, $f'(\textbf{x})$ can't be a vector, derivative of a vector should be a matrix, right? Is this a formal notation for element-wise derivative of vector?


You are partially right and partially wrong: $f'(\textbf{x})$ is a matrix, but $\text{diag}(f'(\textbf{x}))$ means taking the diagonal of that matrix and making a vector out of it.

| improve this answer | |
  • $\begingroup$ That was what I thought when I saw these notation first, and some of article says diag means making vector out of it just like you said. but I saw some note about back-gradient. I saw this formula in stanford cs224n slides ∂h_t/∂h_(t-1) = diag(f'(W_h h_(t-1) + W_x x_t))W_h = I W_h = W_h web.stanford.edu/class/cs224n/slides/… page 12. proceedings.mlr.press/v28/pascanu13.pdf page 2. it says 'where diag converts a vector into a diagonal matrix' and sigma' is element-wise derivative. So I am confused. you mean this is informal? $\endgroup$ – NeverneverNever May 26 at 1:31
  • $\begingroup$ You are right, See and Hewitt's slides and Pascanu's paper use the notation in the sense you quote. But I don't think this is a very common notation---otherwise Pascanu et al. wouldn't need to explain what "diag" does, would they? As far as I can remember, my introductory textbook on Linear Algebra never used "diag" at all. On the other hand, you can look at it as a kind of polymorphism: "diag" applied to a vector produces a diagonal matrix and, applied on a matrix, produces a vector. That's how numpy.diag() works, and, if I remember correctly, MatLab, too. $\endgroup$ – Igor F. May 26 at 7:35
  • $\begingroup$ Thank you for your answer. $\endgroup$ – NeverneverNever May 28 at 1:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.