When we fit any model into a data set for prediction, what exactly happens behind the scenes?

I am learning Regression and I am a little confused about how exactly it fits the classifier or regression into a dataset.

  • 1
    $\begingroup$ Welcome to the site! You estimate the parameters of the model such that some cost function, which measures how well it fits the data, is minimized. The model is usually differentiable so you can do this by gradient descent. $\endgroup$ – Emre Nov 17 '17 at 7:23

Your asking for a layman's explanation, and the phrasing of your question, suggests to me that you are getting tangled up in the semantics, and I totally get that.

So let's let go of the usual terms as much as we can, and speak in concepts.

You have a bunch of observations -- or maybe even only one observation. This is your data. Maybe you are one of the particularly inquisitive ancients, and you notice some sort of relationship between lunar patterns and tides. While tracking this data, you start getting a sense for what sort of forces might be at work that would explain the apparent cause-and-effect relationship. The set of rules that you eventually develop becomes your model. The more correct your rules, the better you will be able to predict the tide tomorrow, next month, and next year.

This concept you've developed -- probably something to do with gravity -- may be even more generally useful. If the general concept is re-used with different numbers, it can also help explain Earth's annual cycle. And why we see the Morning Star (Venus) with certain patterns.

In the cases where we are re-using the concept, we are "fitting the model to the data" -- we are testing whether the set of rules we've developed -- the model -- can explain the apparent forces at work in the observations -- the data -- that we have.

In the usual applications of regression (or any statistical method), we're not trying to formulate the laws of physics, we're trying to find rules that sufficiently govern complex systems -- often human behaviour -- and so we're unlikely to explain observations perfectly. But a model is useful to the extent that it successfully predicts future observations. Rather than waiting for the future, though, we typically perform analysis on part of a data set, and test for predictive power with the remainder of the data set.

A specific instance of regression analysis might be in trying to explain housing prices. There are many many possible factors -- square meterage (footage), number of floors, swimming pool or not, size of property, shape of property, age, likelihood of flooding, distance to neighbours, energy rating, economic conditions, quality of the local school, etc, etc, etc. Which factors are most important? Do one, two, three of the factors explain 95% of the variation in housing prices? Can we come up with an equation that will predict a house's selling price reasonably well, rather than relying on a real estate agent's recommendation -- and having to second-guess what his/her agenda is -- how they might be playing us?

And if you will humour a little editorializing:

There are countless examples of reliance on experts proving to be a mistake (Nate Silver's "The Signal and the Noise", Daniel Kahneman's "Thinking Fast, Thinking Slow", Philip Tetlock's "Superforecasting") -- Data Science is about finding ways to derive insight from the data, rather than experts' opinions. But many data scientists get caught up in the mechanics, and lose sight of the purpose. It's not just about razzle-dazzle infographics. As Hamming puts it: "the purpose of computing is insight, not numbers". So I applaud your effort to seek a layman's explanation, rather than technical speak, to understand regression.

| improve this answer | |

You need to decide which algorithm you use based on your Target Variable(Output, Dependent Variable).

If your Target Variable is Categorical you need to go for Classifier(Binary/Multi), if your Target Variable is Continuous you need to go for Prediction Algorithms (ie: Regression, Decision Tree, etc...), if the data is Time dependent you need to use Forecasting Techniques (Time Series, Croston, etc...)

For example if you are looking at Linear Regression:

Its basic idea is to fit the data in the form of a line. As we remember in our schooling they taught us about Linear Algebra, where you need to find a solution for a line by substituting points in that, if it satisfies, then it is solvable and many other cases. Same way, here if your target variable is Continuous then you use your features as variables. It attempts to model the relationship between two variables by fitting a linear equation to observed data.

Equation of Linear Regression is:

$$Y = a + bX$$

where $X$ is the Explanatory(Features) variable and $Y$ is the Dependent(Target) variable

Do not get confused with Logistic Regression(LR) and other Regression Techniques, LR is not a Regression technique it is a Classifier, people generally get confused with that.

Regarding Classifiers, if the data is Linearly Separable(Yes/No, 1/0) then we can use LR, if they are not Linearly Separable you need to use techniques like SVM.

| improve this answer | |

There is a hole a the wall you want to cover.Let's think about a series questions you would like to ask?

1. What is the shape of the hole? Is it a circle, a square, a ellipse or maybe a triangle? Notice some of those shape is more general than the other, like a square is just a special rectangular.

2. How do you describe the exact size of that hole? what's the diameter of the circle? How long is the size of the square?

What does this has to do with fitting the model?

The first question about the shape of model is an analogy to the problem of model selection. There are lot of different model you can use to fit a model. There is ordinary linear regression where the model can be represented as a straight line. There is also more complex model like polynomials. And like how square is a special rectangular, linear model is just a very special polynomial.

Once we have a particular type of model we have in mind, we need to find a set a numbers that completely describes that model. This the process of fitting your model. In the case of linear regression, the model is a straight line, and the fitting the model is to find which straight line.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.