I am going through the derivation of normal equation for multivariate linear regression. The equation is given by :

$\theta = (X^{T}X)^{-1}X^{T}Y$

The cost function is given by:

$J(\theta) = \frac{1}{2m}(X\theta-Y)^{T}(X\theta-Y)$


$J(\theta) = \frac{1}{2m}(\theta^{T}X^{T}X\theta - 2(X\theta)^{T}Y + Y^{T}Y)$

Differentiating w.r.t $\theta$ and equating to zero

$\frac{dJ(\theta)}{d\theta} = \frac{d}{d\theta}(\theta^{T}X^{T}X\theta)-\frac{d}{d\theta}(2(X\theta)^{T}Y) = 0$

I want to specifically understand the differentiation of the left term:

$\frac{d}{d\theta}(\theta^{T}X^{T}X\theta) = X^{T}X\frac{d}{d\theta}(\theta^{T}\theta) $

$\frac{d}{d\theta}(\theta^{T}\theta) = [\frac{d}{d\theta_1}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2}),\frac{d}{d\theta_2}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2}) ,...., \frac{d}{d\theta_n}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2})]$

$\frac{d}{d\theta}(\theta^{T}\theta) = [2\theta_1,2\theta_2,...,2\theta_n] $

$\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta^{T} $

But the final equation is obtained by using $\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta$

How is $\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta$ and not $2\theta^{T}$


It is basically a matter of convention, which becomes a bit more clear if you write the whole thing in terms of elements, rather than vectors. Consider

$$ \theta^T \theta = \sum_{n=1}^N \theta_n \theta_n = \sum_{n=1}^N \theta_n^2 $$

What is usually meant if you write $\frac{df}{d\theta}$ is that you take the gradient of a scalar $f$, i.e. you get a vector $\frac{df}{d\theta}$ where each element $i$ is the derivative with respect to the respective coordinate $\theta_i$:

$$ \left( \frac{df}{d\theta} \right)_i = \frac{df}{d\theta_i} $$

Let's apply this to $\theta^T\theta$:

$$ \left( \frac{d}{d\theta} \theta^T\theta \right)_i = \frac{d}{d\theta_i} \sum_{n=1}^N \theta_n^2 = 2 \sum_{n=1}^N \theta_n \frac{d\theta_n}{d\theta_i} $$

and since $\frac{d\theta_n}{d\theta_i} = \delta_{i,n}$:

$$ \left( \frac{d}{d\theta} \theta^T\theta \right)_i = 2 \theta_i $$

If you interpret this as columns or rows is pretty much up to you. Commonly we would take $\theta_i$ as the elements of a column vector $\theta$ and for practicality we usually want the gradient to live in the same vector space as our coordinates, so $\left( \frac{df}{d\theta} \right)_i$ would also be a column vector. Hence

$$ \frac{d}{d\theta} \theta^T\theta = 2 \theta $$


This is a matter of convention.

From the wikipedia page of matrix calculus, note that there are two forms in writing down the derivative, the numerator layout (where result is represented as a row) and also the denominator layout (where result is represented as a column).

I think the denominator form is the more common one from my personal experience.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.