53
votes
Accepted
Why do we need XGBoost and Random Forest?
It's easier to start with your second question and then go to the first.
Bagging
Random Forest is a bagging algorithm. It reduces variance.
Say that you have very unreliable models, such as ...
- 3,430
42
votes
Accepted
Is it necessary to normalize data for XGBoost?
Your rationale is indeed correct: decision trees do not require normalization of their inputs; and since XGBoost is essentially an ensemble algorithm comprised of decision trees, it does not require ...
- 2,075
33
votes
Accepted
XGBRegressor vs. xgboost.train huge speed difference?
xgboost.train will ignore parameter n_estimators, while xgboost.XGBRegressor accepts. In <...
- 4,366
33
votes
Accepted
How is a splitting point chosen for continuous variables in decision trees?
In order to come up with a split point, the values are sorted, and the mid-points between adjacent values are evaluated in terms of some metric, usually information gain or gini impurity. For your ...
- 3,950
26
votes
Accepted
Decision trees: leaf-wise (best-first) and level-wise tree traverse
If you grow the full tree, best-first (leaf-wise) and depth-first (level-wise) will result in the same tree. The difference is in the order in which the tree is expanded. Since we don't normally grow ...
- 3,288
21
votes
When should I use Gini Impurity as opposed to Information Gain (Entropy)?
Gini is intended for continuous attributes and Entropy is for attributes that occur in classes
Gini is to minimize misclassification
Entropy is for exploratory analysis
Entropy is a little ...
- 313
21
votes
When should I use Gini Impurity as opposed to Information Gain (Entropy)?
For the case of a variable with two values, appearing with fractions $f$ and $(1-f)$,
the gini and entropy are given by:
$gini = 2*f(1-f)$
$entropy = f*ln\big({1\over f}\big) + (1-f)*ln\big({1\over(1-...
- 219
19
votes
Accepted
How to make a decision tree with both continuous and categorical variables in the dataset?
Decision trees can handle both categorical and numerical variables at the same time as features, there is not any problem in doing that.
Theory
Every split in a decision tree is based on a feature. If ...
- 6,106
17
votes
Accepted
How max_features parameter works in DecisionTreeClassifier?
Max_feature is the number of features to consider each time to make the split decision. Let us say the dimension of your data is 50 and the max_feature is 10, each time you need to find the split, you ...
- 2,019
16
votes
Accepted
Multicollinearity in Decision Tree
Desicion trees make no assumptions on relationships between features. It just constructs splits on single features that improves classification, based on an impurity measure like Gini or entropy. If ...
- 1,527
15
votes
Accepted
Why continuous features are more important than categorical features in decision tree models?
It could be the way that you encode categorical variables.
If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less ...
- 6,370
12
votes
Accepted
What feature engineering is necessary with tree based algorithms?
Feature engineering that I would consider essential for even tree based algorithms are:
Modular arithmetic calculations: e.g. converting a timestamp into day of the week, or time of day. If your ...
- 441
12
votes
Accepted
Using a random forest, would a RandomForest performance be less if I drop the first or the last tree?
The two slightly-smaller models will perform exactly the same, on average. There is no difference baked in to the different trees: "the last tree will be the best trained" is not true. The ...
- 12.2k
10
votes
When should I use Gini Impurity as opposed to Information Gain (Entropy)?
To add upon the fact that there are more or less the same, consider also the fact that:
$$
\begin{split}
\forall \; 0 < u < 1,\; \log (1-u) &= -u - u^2/2 - u^3/3 \, + \, \cdots\\
\forall \; ...
- 201
10
votes
Why do we need XGBoost and Random Forest?
When you build a tree, you need to define some criteria for splitting nodes. These include metrics like Information Gain and Gini Index. Those are heuristic approaches, they are not guaranteed to give ...
- 947
10
votes
Accepted
Does Tensorflow support a Decision Tree Classifier?
Basically I guess TensorFlow does not support decision trees. I quote from here,
This is a big oversimplification, but there are essentially two types of ...
- 14.2k
10
votes
Accepted
Is max_depth in scikit the equivalent of pruning in decision trees?
Is this equivalent of pruning a decision tree?
Though they have similar goals (i.e. placing some restrictions to the model so that it doesn't grow very complex and overfit), ...
- 8,038
10
votes
Does feature selections matter to Decision Tree algorithms?
For ensembles of decision trees, feature selection is generally not that important. During the induction of decision trees, the optimal feature is selected to split the data based on metrics like ...
- 3,950
9
votes
strings as features in decision tree/random forest
2018 Update!
You can create an embedding (dense vector) space for your categorical variables. Many of you are familiar with word2vec and fastext, which embed words in a meaningful dense vector space....
- 819
9
votes
XGBoost for binary classification: choosing the right threshold
You have to decide what you want to maximize.
Classifying by comparing the probability to 0.5 is appropriate if you want to maximize accuracy. It's not appropriate if you want to maximize the f1 ...
- 3,401
9
votes
How to get a confidence score for predictions?
No matter the model, you can always use the non-parametric bootstrap to construct a confidence interval for any parameter, including predictions (which are actually random variables themselves but are ...
- 3,288
9
votes
How to make a decision tree with both continuous and categorical variables in the dataset?
I am not sure if most answers consider the fact that splitting categorical variables is quite complex. Consider a predictor/feature that has "q" possible values, then there are ~ $2^q$ ...
- 91
9
votes
Accepted
I got 100% accuracy on my test set,is there something wrong?
There may be a few reason this is happening.
First of all, check your code. 100% accuracy seems unlikely in any setting. How many testing data points do you have? How many training data points did ...
- 156
9
votes
When does decision tree perform better than the neural network?
Neural Networks, in my experience have several hyper-parameters (number of layers, neurons per layer, activation functions, optimizers, regularizers, etc.) and are very hard in finding the best ...
- 432
9
votes
Why decision tree needs categorical variable to be encoded?
...why is encoding needed on categorical variables?
That isn't true; decision trees can be built on both continuous and categorical features. (Why don't tree ensembles require one-hot-encoding? )...
- 12.2k
9
votes
Accepted
Decision Trees - how does split for categorical features happen?
You are right on all counts:
If DT splits a node with the above algorithm and treat those 10 values are true numeric values, will it not lead to wrong/misinterpreted splits?
Yes absolutely, ...
- 25.9k
8
votes
Accepted
Why `max_features=n_features` does not make the Random Forest independent of number of trees?
Interesting puzzle indeed.
First things first. The DecisionTreeClassifier has some stochastic behavior. For instance, the splitter code iterates through the ...
- 3,430
8
votes
Accepted
Why don't tree ensembles require one-hot-encoding?
The encoding leads to a question of representation and the way that the algorithms cope with the representation.
Let's consider 3 methods of representing n categorial values of a feature:
A single ...
- 2,663
8
votes
Accepted
How to (better) discretize continuous data in decision trees?
No, you probably don't want to try all possible cut points in a serious implementation. That's how we describe it in simple introductions to ID3, because it's easier to understand, but it's typically ...
- 3,401
8
votes
Accepted
Handling outliers and Null values in Decision tree
Outliers: In decision tree learning, you do splits based on a metric that depends on the proportions of the classes on the left and right leaves after the split (for instance, Giny Impurity). If there ...
- 6,106
Only top scored, non community-wiki answers of a minimum length are eligible