I understand Random Forest models can be used both for classification and regression situations.

Is there a more specific criteria to determine where a random forest model would perform better than common regressions (Linear, Lasso, etc) to estimate values or Logistic Regression for classification?

  • $\begingroup$ All depends on your data, check for each model pre-assumption and you’ll have a general Idea of what is best route/model to approach the problem you try to solve. $\endgroup$ – n1tk Jun 30 at 13:54

Adding some extra general points to the previous answer:

  • As a decision tree algorithm, Random Forests are less influenced by outliers than other algorithms. A good discussion about it is here.
  • They also do not make any assumptions about the underlying distribution of your data, and can implicitly handle collinearity in features, because if you have two highly similar features, the information gain from splitting on one of the features will also use up the predictive power of the other feature. Read about it here.
  • Random Forests can be used for feature selection because if you fit the algorithm with features that are not useful, the algorithm simply won't use them to split on the data. It's possible to extract the 'best' features (which could be the total number of times a feature was used to split on the data, or the mean decrease in impurity etc). However, as with my point above, you cannot read too much into the relative importance of the features, especially if you have ones that are highly correlated.
  • I think the biggest deciding factor of whether to use a RF or another algorithm is probably if you want to understand more about the relationship that features have with the target and the degree of influence they have. If this is important for your use case (for example, you want to know if a feature has a positive or negative relationship with the target and the degree to which it affects the outcome) then others like Logistic Regression and Lasso are better choices.
  • Also, if you want your model to extrapolate to predictions for data that is outside of the bounds of your original training data, a Random Forest will not be a good choice.

Hope this helps!

  • $\begingroup$ "Also, if you want your model to extrapolate to predictions for data that is outside of the bounds of your original training data, a Random Forest will not be a good choice." is this true for any (if not all) models? I'm trying to think of a model where this is not the case. $\endgroup$ – Alexander Jun 30 at 14:00
  • 2
    $\begingroup$ Simplistic example, but imagine you are building a house price model and your training data only goes up to houses with 4 beds and a max price of 700,000, but now a house with 10 bedrooms comes onto the market & you want to estimate the price. If you built a linear regression, and the coefficient for number of beds is 150,000 (and ignoring any other features) then 10 * 150,000 = predicted value of 1.5m. However, a Random Forest will have only seen properties valued up to 700,000 and that is the biggest value it would be able to predict. $\endgroup$ – Matt Jun 30 at 14:20
  • $\begingroup$ Ah, I see. The models will both perform poorly via extrapolation, but Random Forest might perform even more poorly. $\endgroup$ – Alexander Jun 30 at 14:22
  • $\begingroup$ Pretty much, yes $\endgroup$ – Matt Jun 30 at 14:23

This is only a general answer but in case it helps:

  • In general decision trees tend to be "robust" in the sense that they can work with practically any kind of data, in particular in cases where other methods such as linear or logistic regression might struggle. For example they have no problem in the case of heterogeneous features, e.g. when mixing categorical and numerical features, or mixing completely different ranges of values, etc.

  • Random forests add ensemble learning to the mix, making decision trees even more robust and especially well equipped to deal with noisy data, whereas standard regression methods can get easily confused by noise.

Intuitively I see decision trees (random forests included) as the "swiss army knife" of supervised learning: efficient, versatile, easy to use.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.