# Normal equation for linear regression

I am going through the derivation of normal equation for multivariate linear regression. The equation is given by :

$$\theta = (X^{T}X)^{-1}X^{T}Y$$

The cost function is given by:

$$J(\theta) = \frac{1}{2m}(X\theta-Y)^{T}(X\theta-Y)$$

Simplifying,

$$J(\theta) = \frac{1}{2m}(\theta^{T}X^{T}X\theta - 2(X\theta)^{T}Y + Y^{T}Y)$$

Differentiating w.r.t $$\theta$$ and equating to zero

$$\frac{dJ(\theta)}{d\theta} = \frac{d}{d\theta}(\theta^{T}X^{T}X\theta)-\frac{d}{d\theta}(2(X\theta)^{T}Y) = 0$$

I want to specifically understand the differentiation of the left term:

$$\frac{d}{d\theta}(\theta^{T}X^{T}X\theta) = X^{T}X\frac{d}{d\theta}(\theta^{T}\theta)$$

$$\frac{d}{d\theta}(\theta^{T}\theta) = [\frac{d}{d\theta_1}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2}),\frac{d}{d\theta_2}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2}) ,...., \frac{d}{d\theta_n}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2})]$$

$$\frac{d}{d\theta}(\theta^{T}\theta) = [2\theta_1,2\theta_2,...,2\theta_n]$$

$$\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta^{T}$$

But the final equation is obtained by using $$\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta$$

How is $$\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta$$ and not $$2\theta^{T}$$

It is basically a matter of convention, which becomes a bit more clear if you write the whole thing in terms of elements, rather than vectors. Consider

$$\theta^T \theta = \sum_{n=1}^N \theta_n \theta_n = \sum_{n=1}^N \theta_n^2$$

What is usually meant if you write $$\frac{df}{d\theta}$$ is that you take the gradient of a scalar $$f$$, i.e. you get a vector $$\frac{df}{d\theta}$$ where each element $$i$$ is the derivative with respect to the respective coordinate $$\theta_i$$:

$$\left( \frac{df}{d\theta} \right)_i = \frac{df}{d\theta_i}$$

Let's apply this to $$\theta^T\theta$$:

$$\left( \frac{d}{d\theta} \theta^T\theta \right)_i = \frac{d}{d\theta_i} \sum_{n=1}^N \theta_n^2 = 2 \sum_{n=1}^N \theta_n \frac{d\theta_n}{d\theta_i}$$

and since $$\frac{d\theta_n}{d\theta_i} = \delta_{i,n}$$:

$$\left( \frac{d}{d\theta} \theta^T\theta \right)_i = 2 \theta_i$$

If you interpret this as columns or rows is pretty much up to you. Commonly we would take $$\theta_i$$ as the elements of a column vector $$\theta$$ and for practicality we usually want the gradient to live in the same vector space as our coordinates, so $$\left( \frac{df}{d\theta} \right)_i$$ would also be a column vector. Hence

$$\frac{d}{d\theta} \theta^T\theta = 2 \theta$$

This is a matter of convention.

From the wikipedia page of matrix calculus, note that there are two forms in writing down the derivative, the numerator layout (where result is represented as a row) and also the denominator layout (where result is represented as a column).

I think the denominator form is the more common one from my personal experience.