0
$\begingroup$

I'm looking through an online course about machine learning and the first big topic is finding a model that approximates our data with linear regression. The model itself is linear function and we look for the coefficients of that linear function. We introduce non-linearities that the data might have into our linear model by adding feature crosses.

For example, lets say we have two-dimensional features ($x_1, x_2$) and label them as $1$ if they are in the 2nd or 4th quadrant, 0 otherwise. A linear function $y(x_1, x_2) = b +m_1x_1+m_2x_2$ cannot approximate this. If we add $x_3 = x_1x_2$, we can find a model that represents that.

However, this is obviously a very artificial problem and not at representative to solve against any kind of data. In the end, we are trying to find a function $y$ that fits some data. This reminds me of solving differential equations numerically. And there we build the function $y$ as a sum of local basis functions $y(x) = \sum_i m_i\phi_i(x)$.

Why would we use global basis functions instead of local basis functions to introduce non-linearity into our linear machine learning model? Why don't we discretize our feature space and optimize on a linear function space?

$\endgroup$

1 Answer 1

1
$\begingroup$

I see a couple of parts to your question

Part 1

Why would we use global basis functions instead of local basis functions to introduce non-linearity into our linear machine learning model?

Local (non-global) basis functions are used sometimes. For example, radial basis functions (RBF) are sometimes used as a basis in linear regression (see these lecture notes).

However, often it matters that a model is interpretable and more complicated models are harder to interpret / draw certain conclusions from. For example, imagine you're doing a controlled experiment, you may not care about "just" predicting some variable from (the variable you're manipulating + the variables you're controlling for). But rather, how much of an effect your manipulation has on the output.

Part 2

Why don't we discretize our feature space and optimize on a linear function space?

RBFs kind-of discretize feature space. People don't usually literally discretized feature space because discrete optimization is much much more complicated than continuous optimization.

$\endgroup$
1
  • $\begingroup$ Thank you! The RBF are exactly the thing I had in mind. But my question remains: why are they only rarely used? Being not ''interpretable'' enough does not strike me as very convincing. Deep neural networks are not interpretable at all but used everywhere. $\endgroup$
    – don-joe
    Sep 12, 2019 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.