I know from calculus that any relatively well-behaved function $y=f(x)$ can be approximated by a linear function $y=ax+b$ within a sufficiently small neighborhood around each point of an independent variable $x$. For a larger neighborhood, I need a quadratic function, then cubic, etc.

As data are often modeled as points in some high-dimensional space where each dimension relates to a particular feature, it is possible to say that any regression task can be modeled by a linear regression for a subset of data that consists of all data points sufficiently close to a particular data instance? Can it be said that a linear regression with products of features is needed for a larger neighborhood, in the analogy with calculus?

Are there any general theoretical results about the behavior of data in the neighborhood of a single data point?

  • $\begingroup$ Theoretically a function can be any map. It does not need to be continuous or have any nice description at all. The other questions are quite broad and it is not clear whether you are interested in a theoretical discussion or you wish to find a better solution to a particular problem. $\endgroup$
    – Valentas
    May 17 at 16:13
  • $\begingroup$ @Valentas I am interested is it possible instead of using complicated things like neural networks, decision trees or random forests, simply divide all data into small neighborhoods and use linear regression for each neighborhood. $\endgroup$ May 18 at 9:34
  • 1
    $\begingroup$ Yes, and classical techniques like simple knn and kernel methods are related to what you propose. When the dimension is small there is a variety of base functions that you can use to approximate well-behaved, for example continuous on a bounded domain, functions. But when the dimension of inputs becomes large there is the "curse of dimensionality" phenomenon - most of the small neighborhoods will be empty. Therefore the modern techniques try to map the data into a smaller dimension embeddings space. $\endgroup$
    – Valentas
    May 18 at 12:02
  • $\begingroup$ @Valentas Thanks. I think, I understand. It looks like you can answer my question instead of just comments $\endgroup$ May 20 at 1:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.