Universal function approximation with fixed values (as vector or matrix)

I was thinking about way to represent/approximate universal function and came up with the idea that a plain fixed numbers could be used to represent pretty much any function on a fixed interval.

I wonder how such approach called. And if it's used in logistic regression, genetic algorithms etc.?

The good thing is that it's universal, you can extend it to multiple dimensions and you don't have to make any assumption about function if it should be polynomial or trigonometric etc.

I would like to know more about it. But can't find much.

2 Answers

What you highlight in this question is actually the boundary between regression (predicting continuous variables) and classification (predicting outcomes). Indeed, predicting discrete numerical variables or ordered categorical outcomes can fall in either regression or classification, depending on how you interpret the problem.

Some algortihms are natively made for classification, and therefore, when used for regression on a continuous target variable, actually produce a prediction on a discrete approximation of the target variable. As highlighted by Ben, decision trees belong to this category. Based on the approach you suggest, almost any multi-class classification algorithm can be used for regression.

Oppositely, algorithms designed for regression can be used for classification purposes by doing the inverse operation. For example, logistic regression, which is a regression algorithm because it predicts a continuous variable, is often used as a classifier when it actually predicts the probability for a given class.

By "plain fixed numbers" do you mean a piecewise-constant function? Sure, those are universal approximators (with some natural hypotheses*). Indeed, decision trees generate a special kind of piecewise-constant function, and they are universal approximators*.

*I've actually been unable to find references for these, and am not sure how to work out reasonable hypotheses. Using the examples $$\sin(x)$$ and $$\sin(1/x)$$ for $$x>0$$ indicates that we need a compact domain, and surely some sort of continuity is required. That's probably enough, but my analysis classes are too long forgotten to know how to produce a proof; you might need some fancier stuff like Stone-Weierstrass. If you take Lipschitz continuity as a hypothesis, it seems like there'd be an easy(/ier) direct argument.

• Thanks, yes piecewise-constant function, I'm mostly interested in practical usage - where and how they are used, decision trees seems to be a good example. – Alexey Petrushin Nov 27 '19 at 21:35
• @AlexeyPetrushin k-nearest-neighbors also produces a piecewise-constant function. – Ben Reiniger Nov 30 '19 at 2:43