Is the gradient descent the same if cost function has interaction?

Question

We know how to determine regression parameters using gradient descent. If

and the cost function is C=|Y-Y(X)|^2, we update b as

where is the learning rate and is the partial differential of the cost function C with respect to b.

If in multiple regression there exist an interaction and we want to stick on the linear model formulation (not using tree or other non-linear regressors), such that

and the cost function is still the same, do we just do the same way to update b? i.e. the existence of interaction terms doesn't have impact on gradient descent. I didn't see any difference of gradient descent between with/without interaction.

Answer 1

The usual way to use interaction terms in linear regression is to construct new $x_n$, e.g. $x_3 = x_1 x_2$, and treat those identically as any other $x_n$. The learned parameter $b$ does not "know" the difference in how you calculated $x$, and the problem is still considered linear regression even if you create really complex functions of $x_n$ to create an input.

Taking your example, but with slightly different notation:

• The model estimate for Y is $\hat{y} = a + bx_1x_2$

• The value of Y you want to learn is $y$

• Mean squared error for a single example is $L = \frac{1}{2}(y-\hat{y})^2$ The factor of 2 does not change this answer, and is commonly used to simplify the gradient. Typically $C$ is the mean of $L$ over all examples.

In order to learn optimal value of $b$, for one example the gradient you need is $\frac{\partial L}{\partial b}$. We can get that by expanding the loss function:

$L = \frac{1}{2}(y-\hat{y})^2$

$L = \frac{1}{2}(y^2 - 2y\hat{y} + \hat{y}^2)$

We could expand further, but typically now we calculate $\frac{\partial L}{\partial \hat{y}}$ and use the chain rule, because that has a simpler, more intuitive-looking result. Terms without $\hat{y}$ are zero:

$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$

We want $\frac{\partial L}{\partial b}$ for gradient descent

$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial b}$ (Chain rule)

$\frac{\partial L}{\partial b} = (\hat{y} - y)(\frac{\partial}{\partial b} a + bx_1x_2)$

Again, terms without $b$ in them are constants:

$\frac{\partial L}{\partial b} = (\hat{y} - y)(x_1x_2)$

Note that the $x_1x_2$ term is unchanged from the input. It could be any function $x_n = f(x_1, x_2, x_3 ....)$