8
$\begingroup$

I am working on a project and I am having difficulty in deciding which algorithm to choose for regression. I want to know under what conditions should one choose a linear regression or Decision Tree regression or Random Forest regression? Are there any specific characteristics of the data that would make the decision to go towards a specific algorithm amongst the tree mentioned above? What are those characteristics that I should look in my dataset to make the decision? And are there some reasons that would make one choose a decision tree or random forest algorithm even if the same correctness can be achieved by linear regression?

$\endgroup$
  • $\begingroup$ This is too broad -- start with a description of your data and what your constraints are? $\endgroup$ – Sean Owen Dec 2 '15 at 14:44
8
$\begingroup$

Let me explain it using some examples for clear intuition:

When do you use linear regression vs Decision Trees?

Linear regression is a linear model, which means it works really nicely when the data has a linear shape. But, when the data has a non-linear shape, then a linear model cannot capture the non-linear features.

So in this case, you can use the decision trees, which do a better job at capturing the non-linearity in the data by dividing the space into smaller sub-spaces depending on the questions asked.

When do you use Random Forest vs Decision Trees?

I guess the Quora answer here would do a better job than me, at explaining the difference between them and their applications. Let me quote that for you:

Suppose you're very indecisive, so whenever you want to watch a movie, you ask your friend Willow if she thinks you'll like it. In order to answer, Willow first needs to figure out what movies you like, so you give her a bunch of movies and tell her whether you liked each one or not (i.e., you give her a labeled training set). Then, when you ask her if she thinks you'll like movie X or not, she plays a 20 questions-like game with IMDB, asking questions like "Is X a romantic movie?", "Does Johnny Depp star in X?", and so on. She asks more informative questions first (i.e., she maximizes the information gain of each question), and gives you a yes/no answer at the end.

Thus, Willow is a decision tree for your movie preferences.

But Willow is only human, so she doesn't always generalize your preferences very well (i.e., she overfits). In order to get more accurate recommendations, you'd like to ask a bunch of your friends, and watch movie X if most of them say they think you'll like it. That is, instead of asking only Willow, you want to ask Woody, Apple, and Cartman as well, and they vote on whether you'll like a movie (i.e., you build an ensemble classifier, aka a forest in this case).

Now you don't want each of your friends to do the same thing and give you the same answer, so you first give each of them slightly different data. After all, you're not absolutely sure of your preferences yourself -- you told Willow you loved Titanic, but maybe you were just happy that day because it was your birthday, so maybe some of your friends shouldn't use the fact that you liked Titanic in making their recommendations. Or maybe you told her you loved Cinderella, but actually you really really loved it, so some of your friends should give Cinderella more weight. So instead of giving your friends the same data you gave Willow, you give them slightly perturbed versions. You don't change your love/hate decisions, you just say you love/hate some movies a little more or less (you give each of your friends a bootstrapped version of your original training data). For example, whereas you told Willow that you liked Black Swan and Harry Potter and disliked Avatar, you tell Woody that you liked Black Swan so much you watched it twice, you disliked Avatar, and don't mention Harry Potter at all.

By using this ensemble, you hope that while each of your friends gives somewhat idiosyncratic recommendations (Willow thinks you like vampire movies more than you do, Woody thinks you like Pixar movies, and Cartman thinks you just hate everything), the errors get canceled out in the majority. Thus, your friends now form a bagged (bootstrap aggregated) forest of your movie preferences.

There's still one problem with your data, however. While you loved both Titanic and Inception, it wasn't because you like movies that star Leonardio DiCaprio. Maybe you liked both movies for other reasons. Thus, you don't want your friends to all base their recommendations on whether Leo is in a movie or not. So when each friend asks IMDB a question, only a random subset of the possible questions is allowed (i.e., when you're building a decision tree, at each node you use some randomness in selecting the attribute to split on, say by randomly selecting an attribute or by selecting an attribute from a random subset). This means your friends aren't allowed to ask whether Leonardo DiCaprio is in the movie whenever they want. So whereas previously you injected randomness at the data level, by perturbing your movie preferences slightly, now you're injecting randomness at the model level, by making your friends ask different questions at different times.

And so your friends now form a random forest.

$\endgroup$
  • 5
    $\begingroup$ when the data has a non-linear shape, then a linear model cannot capture the non-linear features This is a common misconception. First of all, a simple linear regression can model even harmonic series stats.stackexchange.com/questions/60500/…. Secondly, feature interaction can be introduced and, of course, there are generalized linear model where a non-linear function on the linear terms is introduced (for instance, the logistic regression). $\endgroup$ – Ricardo Cruz May 30 '16 at 14:12
2
$\begingroup$

As far as I know, there is not a rule to say which algorithm works for which dataset. Just make sure your dataset and variables of interest fulfill the pre-assumptions of running each algorithm and give it a try. For example, linear regression has some pre-assumptions such as normality of resuduals, homoscedasticity (the variability in the response variable is the same at all levels of the explanatory variable) and so on. Just check these for your variables and give the algorithm a try.

You can use a point and click software to see the results without getting involved in the code and parameter setting. If you are an R user, rattle package will be a very useful tool at this stage. You do your job in point and click mode and you have access to the code behind it.

$\endgroup$
  • $\begingroup$ The only rule of thumb I have read is that regressions handle noise better than random forests, which sounds true because decision trees are discrete models, but I never saw this quantitatively tested. $\endgroup$ – Ricardo Cruz May 30 '16 at 14:14

Not the answer you're looking for? Browse other questions tagged or ask your own question.