# Is the gradient descent the same if cost function has interaction?

We know how to determine regression parameters using gradient descent. If

$Y = a + bx_1$

and the cost function is C=|Y-Y(X)|^2, we update b as

$b := b - \alpha \frac{\partial C}{\partial b}$

where $\alpha$ is the learning rate and $\partial C/\partial b$ is the partial differential of the cost function C with respect to b.

If in multiple regression there exist an interaction and we want to stick on the linear model formulation (not using tree or other non-linear regressors), such that

$Y = a + b x_1 x_2$

and the cost function is still the same, do we just do the same way to update b? i.e. the existence of interaction terms doesn't have impact on gradient descent. I didn't see any difference of gradient descent between with/without interaction.

The usual way to use interaction terms in linear regression is to construct new $x_n$, e.g. $x_3 = x_1 x_2$, and treat those identically as any other $x_n$. The learned parameter $b$ does not "know" the difference in how you calculated $x$, and the problem is still considered linear regression even if you create really complex functions of $x_n$ to create an input.

Taking your example, but with slightly different notation:

• The model estimate for Y is $\hat{y} = a + bx_1x_2$

• The value of Y you want to learn is $y$

• Mean squared error for a single example is $L = \frac{1}{2}(y-\hat{y})^2$ The factor of 2 does not change this answer, and is commonly used to simplify the gradient. Typically $C$ is the mean of $L$ over all examples.

In order to learn optimal value of $b$, for one example the gradient you need is $\frac{\partial L}{\partial b}$. We can get that by expanding the loss function:

$L = \frac{1}{2}(y-\hat{y})^2$

$L = \frac{1}{2}(y^2 - 2y\hat{y} + \hat{y}^2)$

We could expand further, but typically now we calculate $\frac{\partial L}{\partial \hat{y}}$ and use the chain rule, because that has a simpler, more intuitive-looking result. Terms without $\hat{y}$ are zero:

$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$

We want $\frac{\partial L}{\partial b}$ for gradient descent

$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial b}$ (Chain rule)

$\frac{\partial L}{\partial b} = (\hat{y} - y)(\frac{\partial}{\partial b} a + bx_1x_2)$

Again, terms without $b$ in them are constants:

$\frac{\partial L}{\partial b} = (\hat{y} - y)(x_1x_2)$

Note that the $x_1x_2$ term is unchanged from the input. It could be any function $x_n = f(x_1, x_2, x_3 ....)$