# when x is a vector, derivative of vector diag(f'(x)) is formal notation?

(4)

this note says this

$$\frac{\partial \textbf{z}}{\partial \textbf{x}} = \text{diag}(f'(\textbf{x}))$$

I know this means make a vector to square matrix which has an original vector as diagonal component.

but, since $$\textbf{x}$$ is a vector, $$f'(\textbf{x})$$ can't be a vector, derivative of a vector should be a matrix, right? Is this a formal notation for element-wise derivative of vector?

You are partially right and partially wrong: $$f'(\textbf{x})$$ is a matrix, but $$\text{diag}(f'(\textbf{x}))$$ means taking the diagonal of that matrix and making a vector out of it.

• That was what I thought when I saw these notation first, and some of article says diag means making vector out of it just like you said. but I saw some note about back-gradient. I saw this formula in stanford cs224n slides ∂h_t/∂h_(t-1) = diag(f'(W_h h_(t-1) + W_x x_t))W_h = I W_h = W_h web.stanford.edu/class/cs224n/slides/… page 12. proceedings.mlr.press/v28/pascanu13.pdf page 2. it says 'where diag converts a vector into a diagonal matrix' and sigma' is element-wise derivative. So I am confused. you mean this is informal? – NeverneverNever May 26 at 1:31
• You are right, See and Hewitt's slides and Pascanu's paper use the notation in the sense you quote. But I don't think this is a very common notation---otherwise Pascanu et al. wouldn't need to explain what "diag" does, would they? As far as I can remember, my introductory textbook on Linear Algebra never used "diag" at all. On the other hand, you can look at it as a kind of polymorphism: "diag" applied to a vector produces a diagonal matrix and, applied on a matrix, produces a vector. That's how numpy.diag() works, and, if I remember correctly, MatLab, too. – Igor F. May 26 at 7:35
• Thank you for your answer. – NeverneverNever May 28 at 1:14