3
$\begingroup$

I understand Random Forest models can be used both for classification and regression situations.

Is there a more specific criteria to determine where a random forest model would perform better than common regressions (Linear, Lasso, etc) to estimate values or Logistic Regression for classification?

$\endgroup$
  • $\begingroup$ All depends on your data, check for each model pre-assumption and you’ll have a general Idea of what is best route/model to approach the problem you try to solve. $\endgroup$ – n1tk Jun 30 '19 at 13:54
3
$\begingroup$

Adding some extra general points to the previous answer:

  • As a decision tree algorithm, Random Forests are less influenced by outliers than other algorithms. A good discussion about it is here.
  • They also do not make any assumptions about the underlying distribution of your data, and can implicitly handle collinearity in features, because if you have two highly similar features, the information gain from splitting on one of the features will also use up the predictive power of the other feature. Read about it here.
  • Random Forests can be used for feature selection because if you fit the algorithm with features that are not useful, the algorithm simply won't use them to split on the data. It's possible to extract the 'best' features (which could be the total number of times a feature was used to split on the data, or the mean decrease in impurity etc). However, as with my point above, you cannot read too much into the relative importance of the features, especially if you have ones that are highly correlated.
  • I think the biggest deciding factor of whether to use a RF or another algorithm is probably if you want to understand more about the relationship that features have with the target and the degree of influence they have. If this is important for your use case (for example, you want to know if a feature has a positive or negative relationship with the target and the degree to which it affects the outcome) then others like Logistic Regression and Lasso are better choices.
  • Also, if you want your model to extrapolate to predictions for data that is outside of the bounds of your original training data, a Random Forest will not be a good choice.

Hope this helps!

| improve this answer | |
$\endgroup$
  • $\begingroup$ "Also, if you want your model to extrapolate to predictions for data that is outside of the bounds of your original training data, a Random Forest will not be a good choice." is this true for any (if not all) models? I'm trying to think of a model where this is not the case. $\endgroup$ – Alexander Jun 30 '19 at 14:00
  • 2
    $\begingroup$ Simplistic example, but imagine you are building a house price model and your training data only goes up to houses with 4 beds and a max price of 700,000, but now a house with 10 bedrooms comes onto the market & you want to estimate the price. If you built a linear regression, and the coefficient for number of beds is 150,000 (and ignoring any other features) then 10 * 150,000 = predicted value of 1.5m. However, a Random Forest will have only seen properties valued up to 700,000 and that is the biggest value it would be able to predict. $\endgroup$ – Matt Jun 30 '19 at 14:20
  • $\begingroup$ Ah, I see. The models will both perform poorly via extrapolation, but Random Forest might perform even more poorly. $\endgroup$ – Alexander Jun 30 '19 at 14:22
  • $\begingroup$ Pretty much, yes $\endgroup$ – Matt Jun 30 '19 at 14:23
1
$\begingroup$

This is only a general answer but in case it helps:

  • In general decision trees tend to be "robust" in the sense that they can work with practically any kind of data, in particular in cases where other methods such as linear or logistic regression might struggle. For example they have no problem in the case of heterogeneous features, e.g. when mixing categorical and numerical features, or mixing completely different ranges of values, etc.

  • Random forests add ensemble learning to the mix, making decision trees even more robust and especially well equipped to deal with noisy data, whereas standard regression methods can get easily confused by noise.

Intuitively I see decision trees (random forests included) as the "swiss army knife" of supervised learning: efficient, versatile, easy to use.

| improve this answer | |
$\endgroup$
1
$\begingroup$

Please consider the following:

1) Random forest algorithm can be used for both classifications and regression task.
2) It typically provides very high accuracy.
3) Random forest classifier will handle the missing values and maintain the accuracy of a large proportion of data.
4) If there are more trees, it usually won’t allow overfitting trees in the model.
5) It has the power to handle a large data set with high dimensionality

Ultimately, what algo you choose to work with is up to you. you definitely want the predictive capabilities of the algo to be pretty high (over 90%). Sometimes other algos beat the RF algo, but I have found that often the RF is quite good! Usually, I start with RF, and if I see decent performance, I am done. I believe, in around at least 80% of the time, I'm done. If you are not getting good results from the RF algo, test some others.

This gives a nice comparison of a few different algos.

import pandas
import matplotlib.pyplot as plt
from sklearn import model_selection
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
# load dataset
url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = pandas.read_csv(url, names=names)
array = dataframe.values
X = array[:,0:8]
Y = array[:,8]
# prepare configuration for cross validation test harness
seed = 7
# prepare models
models = []
models.append(('LR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC()))
# evaluate each model in turn
results = []
names = []
scoring = 'accuracy'
for name, model in models:
    kfold = model_selection.KFold(n_splits=10, random_state=seed)
    cv_results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
    print(msg)
# boxplot algorithm comparison
fig = plt.figure()
fig.suptitle('Algorithm Comparison')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()

https://machinelearningmastery.com/compare-machine-learning-algorithms-python-scikit-learn/

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.