Skip to main content
Share Your Experience: Take the 2024 Developer Survey
52 votes
Accepted

Why do we need XGBoost and Random Forest?

It's easier to start with your second question and then go to the first. Bagging Random Forest is a bagging algorithm. It reduces variance. Say that you have very unreliable models, such as ...
Ricardo Cruz's user avatar
  • 3,420
41 votes
Accepted

Is it necessary to normalize data for XGBoost?

Your rationale is indeed correct: decision trees do not require normalization of their inputs; and since XGBoost is essentially an ensemble algorithm comprised of decision trees, it does not require ...
desertnaut's user avatar
  • 2,018
33 votes
Accepted

XGBRegressor vs. xgboost.train huge speed difference?

xgboost.train will ignore parameter n_estimators, while xgboost.XGBRegressor accepts. In <...
Icyblade's user avatar
  • 4,336
33 votes
Accepted

How is a splitting point chosen for continuous variables in decision trees?

In order to come up with a split point, the values are sorted, and the mid-points between adjacent values are evaluated in terms of some metric, usually information gain or gini impurity. For your ...
timleathart's user avatar
  • 3,940
25 votes
Accepted

How to normalize data for Neural Network and Decision Forest

I disagree with the other comments. First of all, I see no need to normalize data for decision trees. Decision trees work by calculating a score (usually entropy) for each different division of the ...
Ricardo Cruz's user avatar
  • 3,420
24 votes
Accepted

Decision trees: leaf-wise (best-first) and level-wise tree traverse

If you grow the full tree, best-first (leaf-wise) and depth-first (level-wise) will result in the same tree. The difference is in the order in which the tree is expanded. Since we don't normally grow ...
David Marx's user avatar
  • 3,258
22 votes

How to predict probabilities in xgboost using R?

Just use predict_proba instead of predict. You can leave the objective as binary:logistic.
ihadanny's user avatar
  • 1,357
21 votes

When should I use Gini Impurity as opposed to Information Gain (Entropy)?

Gini is intended for continuous attributes and Entropy is for attributes that occur in classes Gini is to minimize misclassification Entropy is for exploratory analysis Entropy is a little ...
NIMISHAN's user avatar
  • 313
21 votes

When should I use Gini Impurity as opposed to Information Gain (Entropy)?

For the case of a variable with two values, appearing with fractions $f$ and $(1-f)$, the gini and entropy are given by: $gini = 2*f(1-f)$ $entropy = f*ln\big({1\over f}\big) + (1-f)*ln\big({1\over(1-...
DanLvii Dewey's user avatar
19 votes
Accepted

How to make a decision tree with both continuous and categorical variables in the dataset?

Decision trees can handle both categorical and numerical variables at the same time as features, there is not any problem in doing that. Theory Every split in a decision tree is based on a feature. If ...
David Masip's user avatar
  • 6,081
17 votes
Accepted

How max_features parameter works in DecisionTreeClassifier?

Max_feature is the number of features to consider each time to make the split decision. Let us say the dimension of your data is 50 and the max_feature is 10, each time you need to find the split, you ...
Bashar Haddad's user avatar
16 votes
Accepted

Multicollinearity in Decision Tree

Desicion trees make no assumptions on relationships between features. It just constructs splits on single features that improves classification, based on an impurity measure like Gini or entropy. If ...
Jon Nordby's user avatar
  • 1,527
15 votes
Accepted

Why continuous features are more important than categorical features in decision tree models?

It could be the way that you encode categorical variables. If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less ...
Carlos Mougan's user avatar
12 votes
Accepted

What feature engineering is necessary with tree based algorithms?

Feature engineering that I would consider essential for even tree based algorithms are: Modular arithmetic calculations: e.g. converting a timestamp into day of the week, or time of day. If your ...
Eumenedies's user avatar
12 votes
Accepted

Using a random forest, would a RandomForest performance be less if I drop the first or the last tree?

The two slightly-smaller models will perform exactly the same, on average. There is no difference baked in to the different trees: "the last tree will be the best trained" is not true. The ...
Ben Reiniger's user avatar
  • 11.9k
11 votes
Accepted

Decision tree, how to understand or calculate the probability/confidence of prediction result

What data mining package do you use? In sklearn, the DecisionTreeClassifier can give you probabilities, but you have to use things like max_depth in order to ...
Ricardo Cruz's user avatar
  • 3,420
10 votes

When should I use Gini Impurity as opposed to Information Gain (Entropy)?

To add upon the fact that there are more or less the same, consider also the fact that: $$ \begin{split} \forall \; 0 < u < 1,\; \log (1-u) &= -u - u^2/2 - u^3/3 \, + \, \cdots\\ \forall \; ...
ClementWalter's user avatar
10 votes

Decision tree vs. KNN

Classifiers like Decision Tree, Bayesian, Back-propagation, Support Vector Machine come under the category of "Eager Learners", because they first build a classification model on the training dataset ...
spkakkar's user avatar
  • 216
10 votes
Accepted

First steps with Python and scikit-learn

In python 3 the print function must have parenthesis, so print(clf.predict([[150, 0]])) will work
hipoglucido's user avatar
  • 1,170
10 votes

Why do we need XGBoost and Random Forest?

When you build a tree, you need to define some criteria for splitting nodes. These include metrics like Information Gain and Gini Index. Those are heuristic approaches, they are not guaranteed to give ...
Mephy's user avatar
  • 937
10 votes
Accepted

Does Tensorflow support a Decision Tree Classifier?

Basically I guess TensorFlow does not support decision trees. I quote from here, This is a big oversimplification, but there are essentially two types of ...
Green Falcon's user avatar
  • 14.1k
10 votes
Accepted

Is max_depth in scikit the equivalent of pruning in decision trees?

Is this equivalent of pruning a decision tree? Though they have similar goals (i.e. placing some restrictions to the model so that it doesn't grow very complex and overfit), ...
Djib2011's user avatar
  • 7,998
10 votes

Does feature selections matter to Decision Tree algorithms?

For ensembles of decision trees, feature selection is generally not that important. During the induction of decision trees, the optimal feature is selected to split the data based on metrics like ...
timleathart's user avatar
  • 3,940
9 votes

strings as features in decision tree/random forest

2018 Update! You can create an embedding (dense vector) space for your categorical variables. Many of you are familiar with word2vec and fastext, which embed words in a meaningful dense vector space....
Pete's user avatar
  • 819
9 votes

XGBoost for binary classification: choosing the right threshold

You have to decide what you want to maximize. Classifying by comparing the probability to 0.5 is appropriate if you want to maximize accuracy. It's not appropriate if you want to maximize the f1 ...
D.W.'s user avatar
  • 3,371
9 votes

How to get a confidence score for predictions?

No matter the model, you can always use the non-parametric bootstrap to construct a confidence interval for any parameter, including predictions (which are actually random variables themselves but are ...
David Marx's user avatar
  • 3,258
9 votes

How to make a decision tree with both continuous and categorical variables in the dataset?

I am not sure if most answers consider the fact that splitting categorical variables is quite complex. Consider a predictor/feature that has "q" possible values, then there are ~ $2^q$ ...
seeker's user avatar
  • 91
9 votes
Accepted

I got 100% accuracy on my test set,is there something wrong?

There may be a few reason this is happening. First of all, check your code. 100% accuracy seems unlikely in any setting. How many testing data points do you have? How many training data points did ...
c zl's user avatar
  • 156
9 votes

When does decision tree perform better than the neural network?

Neural Networks, in my experience have several hyper-parameters (number of layers, neurons per layer, activation functions, optimizers, regularizers, etc.) and are very hard in finding the best ...
JkBk's user avatar
  • 432
9 votes

Why decision tree needs categorical variable to be encoded?

...why is encoding needed on categorical variables? That isn't true; decision trees can be built on both continuous and categorical features. (Why don't tree ensembles require one-hot-encoding? )...
Ben Reiniger's user avatar
  • 11.9k

Only top scored, non community-wiki answers of a minimum length are eligible