Skip to main content

Questions tagged [decision-trees]

A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm.

Filter by
Sorted by
Tagged with
112 votes
9 answers
139k views

When should I use Gini Impurity as opposed to Information Gain (Entropy)?

Can someone practically explain the rationale behind Gini impurity vs Information gain (based on Entropy)? Which metric is better to use in different scenarios while using decision trees?
Krish Mahajan's user avatar
87 votes
6 answers
153k views

strings as features in decision tree/random forest

I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, scikit-...
user3001408's user avatar
  • 1,005
40 votes
4 answers
38k views

Why do we need XGBoost and Random Forest?

I wasn't clear on couple of concepts: XGBoost converts weak learners to strong learners. What's the advantage of doing this ? Combining many weak learners instead of just using a single tree ? ...
John Constantine's user avatar
38 votes
5 answers
59k views

Are decision tree algorithms linear or nonlinear

Recently a friend of mine was asked whether decision tree algorithms are linear or nonlinear algorithms in an interview. I tried to look for answers to this question but couldn't find any satisfactory ...
user2966197's user avatar
36 votes
3 answers
37k views

Is it necessary to normalize data for XGBoost?

MinMaxScaler() in scikit-learn is used for data normalization (a.k.a feature scaling). Data normalization is not necessary for ...
user781486's user avatar
  • 1,425
31 votes
1 answer
33k views

How is a splitting point chosen for continuous variables in decision trees?

I have two questions related to decision trees: If we have a continuous attribute, how do we choose the splitting value? Example: Age=(20,29,50,40....) Imagine that we have a continuous attribute $f$...
WALID BELRHALMIA's user avatar
26 votes
4 answers
97k views

How to make a decision tree with both continuous and categorical variables in the dataset?

Let's say I have 3 categorical and 2 continuous attributes in a dataset. How do I build a decision tree using these 5 variables? Edit: For categorical variables, it is easy to say that we will split ...
Sahil Chaturvedi's user avatar
25 votes
4 answers
79k views

How to predict probabilities in xgboost using R?

The below predict function is giving -ve values as well so it cannot be probabilities. ...
GeorgeOfTheRF's user avatar
24 votes
1 answer
19k views

Decision trees: leaf-wise (best-first) and level-wise tree traverse

Issue 1: I am confused by the description of LightGBM regarding the way the tree is expanded. They state: Most decision tree learning algorithms grow tree by level (depth)-wise, like the ...
kkk's user avatar
  • 463
22 votes
1 answer
33k views

XGBRegressor vs. xgboost.train huge speed difference?

If I train my model using the following code: ...
user1566200's user avatar
17 votes
5 answers
51k views

Should I use a decision tree or logistic regression for classification?

I am working on a classification problem. I have a dataset containing equal numbers of categorical variables and continuous variables. How do I decide which technique to use, between a decision tree ...
Arun's user avatar
  • 717
17 votes
5 answers
46k views

Decision tree vs. KNN

In which cases is it better to use a Decision tree and other cases a KNN? Why use one of them in certain cases? And the other in different cases? (By looking at its functionality, not at the ...
gchavez1's user avatar
  • 173
16 votes
2 answers
80k views

When to choose linear regression or Decision Tree or Random Forest regression? [closed]

I am working on a project and I am having difficulty in deciding which algorithm to choose for regression. I want to know under what conditions should one choose a <...
Jason Donnald's user avatar
15 votes
1 answer
719 views

Can gradient boosted trees fit any function?

For neural networks we have the universal approximation theorem which states that neural networks can approximate any continuous function on a compact subset of $R^n$. Is there a similar result for ...
Imran's user avatar
  • 2,381
13 votes
2 answers
30k views

How to normalize data for Neural Network and Decision Forest

I have a data set with 20000 samples, each has 12 different features. Each sample is either in category 0 or 1. I want to train a neural network and a decision forest to categorize the samples so that ...
Merlin1896's user avatar
13 votes
1 answer
22k views

How max_features parameter works in DecisionTreeClassifier?

What is the parameter max_features in DecisionTreeClassifier responsible for? I thought it defines the number of features the ...
James Flash's user avatar
13 votes
3 answers
24k views

Unbalanced classes -- How to minimize false negatives?

I have a dataset that has a binary class attribute. There are 623 instances with class +1 (cancer positive) and 101,671 instances with class -1 (cancer negative). I've tried various algorithms (Naive ...
user798275's user avatar
13 votes
4 answers
20k views

Interpreting Decision Tree in context of feature importances

I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the ...
Tim Lindsey's user avatar
12 votes
1 answer
7k views

What feature engineering is necessary with tree based algorithms?

I understand data hygiene, which is probably the most basic feature engineering. That is making sure all your data is properly loaded, making sure N/As are treated ...
William Entriken's user avatar
12 votes
2 answers
6k views

Why neural networks do not perform well on structured data?

I was recently working on some classification problem where decision trees performed better than neural networks. I had tried various combinations with neural networks altering the number of neurons / ...
Suhail Gupta's user avatar
12 votes
1 answer
32k views

XGBoost for binary classification: choosing the right threshold

I am working on a highly-imbalanced binary-labeled dataset, where number of true labels is just 7% from the whole dataset. But some combination of features could yield higher than average number of ...
Denis Kulagin's user avatar
12 votes
2 answers
8k views

Catboost Categorical Features Handling Options (CTR settings)?

I am working with a dataset with large number of categorical features (>80%) predicting a continuous target variable (i.e. Regression). I have been reading quite a bit about ways to handle categorical ...
TwinPenguins's user avatar
  • 4,279
11 votes
9 answers
48k views

I got 100% accuracy on my test set,is there something wrong?

I got 100% accuracy on my test set when trained using decision tree algorithm.but only got 85% accuracy on random forest Is there something wrong with my model or is decision tree best suited for the ...
Harigovind Valsakumar's user avatar
11 votes
2 answers
4k views

Is max_depth in scikit the equivalent of pruning in decision trees?

I was analyzing the classifier created using a decision tree. There is a tuning parameter called max_depth in scikit's decision tree. Is this equivalent of pruning a decision tree? If not, how could I ...
Suhail Gupta's user avatar
11 votes
1 answer
30k views

Decision tree, how to understand or calculate the probability/confidence of prediction result

For example, a drug prediction problem using a decision tree. I trained the decision tree model and would like to predict using new data. For example: ...
GoingMyWay's user avatar
11 votes
3 answers
3k views

Can regression trees predict continuously?

Suppose I have a smooth function like $f(x, y) = x^2+y^2$. I have a training set $D \subsetneq \{((x, y), f(x,y)) | (x,y) \in \mathbb{R}^2\}$ and, of course, I don't know $f$ although I can evaluate $...
Martin Thoma's user avatar
10 votes
2 answers
20k views

Multicollinearity in Decision Tree

Can anybody please explain the affect of multicollinearity on Decision Tree algorithms (Classification and regression). I have done some searching but was not able to find the right answer as some say ...
deepguy's user avatar
  • 1,441
10 votes
3 answers
6k views

Decision Trees - how does split for categorical features happen?

A decision tree, while performing recursive binary splitting, selects an independent variable (say $X_j$) and a threshold (say $t$) such that the predictor space is split into regions {$X|X_j < t$} ...
Supratim Haldar's user avatar
10 votes
3 answers
17k views

Does Tensorflow support a Decision Tree Classifier?

I am trying to implement decision tree classifier to classify my data set. I am using Python. Now it is easy to implement in scikit learn, but how can I implement this in tensorflow.
Taimur Islam's user avatar
10 votes
2 answers
10k views

Why `max_features=n_features` does not make the Random Forest independent of number of trees?

Consider the following simple classification problem (Python, scikit-learn) ...
Jorge Leitao's user avatar
10 votes
5 answers
14k views

Why decision tree needs categorical variable to be encoded?

As per my intuition, decision trees should work better with categorical variables than with continuous variables. If this is the case, why is encoding needed on categorical variables? Can someone give ...
Mukesh K's user avatar
  • 101
10 votes
2 answers
3k views

How are samples selected from training data in Xgboost

In Random Forest, each tree is not fed with the full batch of training data, only a sample. How does this work for Xgboost? If this sampling happens as well, how does it work for this ML algorithm?
Aman Raparia's user avatar
10 votes
2 answers
18k views

How does class_weight work in Decision Tree

The scikit-learn implementation of DecisionTreeClassifier has a parameter as class_weight. ...
Supratim Haldar's user avatar
10 votes
3 answers
5k views

LightGBM - Why Exclusive Feature Bundling (EFB)?

I'm currently studying GBDT and started reading LightGBM's research paper. In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
Tom's user avatar
  • 103
9 votes
2 answers
2k views

Why continuous features are more important than categorical features in decision tree models?

I have both categorical and continuous features in my prediction model and want to select (and rank) most important features. I have converted all categorical variables into dummy variables using one ...
Shahab Kazemi's user avatar
9 votes
5 answers
3k views

Decision tree with final decision being a linear regression

Question: I want to implement a decision tree with each leaf being a linear regression, does such a model exist (preferable in sklearn)? Example case 1: Mockup data is generated using the formula: <...
Nathan's user avatar
  • 193
9 votes
1 answer
6k views

what is the difference between "fully developed decision trees" and "shallow decision trees"?

As reading Ensemble methods on scikit-learn docs, it says that bagging methods work best with strong and complex models (e.g., fully developed decision trees), in contrast with boosting methods ...
Mithril's user avatar
  • 383
9 votes
2 answers
4k views

XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

So I understand the main difference between Random Forests and GB Methods. Random Forests grow parallel trees and GB Methods grow one tree for each iteration. However, I am confused on the vocab used ...
Jack Armstrong's user avatar
9 votes
2 answers
21k views

Why don't tree ensembles require one-hot-encoding?

I know that models such as random forest and boosted trees don't require one-hot encoding for predictor levels, but I don't really get why. If the tree is making a split in the feature space, then isn'...
moefasa's user avatar
  • 93
9 votes
2 answers
14k views

Does feature selections matter to Decision Tree algorithms?

Background: Currently I'm working on my thesis project, which is to build Tree-based ensemble methods for classification on a large data set. Before I started with modeling, I've spent a large amount ...
Ping 's user avatar
  • 91
9 votes
1 answer
5k views

When does decision tree perform better than the neural network?

I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree ...
Suhail Gupta's user avatar
9 votes
1 answer
12k views

How to get a confidence score for predictions?

In a regression problem, is it possible to calculate a confidence/reliability score for a certain prediction given models like XGBoost or Neural Networks?
Henrique Nader's user avatar
9 votes
1 answer
3k views

Minimum number of trees for Random Forest classifier

I am searching for a theoretical or experimental estimation of the lower bound for the number of trees in a Random Forest classifier. I usually test different combinations and select the one that (...
gc5's user avatar
  • 879
8 votes
3 answers
5k views

R vs. Python Decision Tree

From my experiences the R Decision tree returns more accurate results than the python decision tree. Can anymore confirm this assumption and maybe knows the reason?
Rene B.'s user avatar
  • 369
8 votes
1 answer
3k views

How to (better) discretize continuous data in decision trees?

Standard decision tree algorithms, such as ID3 and C4.5, have a brute force approach for choosing the cut point in a continuous feature. Every single value is tested as a possible cut point. (By ...
AutoMiner's user avatar
  • 169
8 votes
2 answers
26k views

How to interpret a decision tree correctly?

I'm trying to work out if I'm correctly interpreting a decision tree found online. The dependent variable of this decision tree is Credit Rating which has two classes, Bad or Good. The root of this ...
DataD's user avatar
  • 83
8 votes
2 answers
10k views

Is there any way to get samples in under each leaf of a decision tree in Sklearn ?

I have trained decision tree . I also have a graph of the tree ( ) . Now i want to see which samples (red circled ones ) are under which leafs . I am using sklearn's implantation . Is there any way ...
Farshid Rayhan's user avatar
8 votes
2 answers
7k views

Can a decision tree learn to solve a xOR problem?

I have read online that decision trees can solve xOR type problems, as shown in images (xOR problem: 1) and (Possible solution as decision tree: 2). My question is how can a decision tree learn to ...
lguerra's user avatar
  • 83
8 votes
1 answer
1k views

Why gradient boosting uses sampling without replacement?

In Random Forest each tree is built selecting a sample with replacement (bootstrap). And I assumed that Gradient Boosting's trees were selected with the same sampling technique. (@BenReiniger ...
Carlos Mougan's user avatar
8 votes
2 answers
12k views

Information Gain in R

I found packages being used to calculating "Information Gain" for selecting main attributes in C4.5 Decision Tree and I tried using them to calculating "Information Gain". But the results of ...
Archimpressom's user avatar

1
2 3 4 5
16