Questions tagged [decision-trees]

A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm.

Filter by
Sorted by
Tagged with
8
votes
4answers
6k views

Why decision tree needs categorical variable to be encoded?

As per my intuition, decision trees should work better with categorical variables than with continuous variables. If this is the case, why is encoding needed on categorical variables? Can someone give ...
26
votes
1answer
26k views

How is a splitting point chosen for continuous variables in decision trees?

I have two questions related to decision trees: If we have a continuous attribute, how do we choose the splitting value? Example: Age=(20,29,50,40....) Imagine that we have a continuous attribute $f$...
5
votes
2answers
712 views

Is Label Encoding with arbitrary numbers ever useful at all?

From what I read online, there seems to be some confusion regarding the taxonomy and the terms used, so to avoid misunderstanding I'm going to define them here: Label Encoding - encoding a nominal ...
2
votes
1answer
243 views

Can a decision in a node of a decision tree be based on comparison between 2 columns of the dataset?

Assume the features in the dataframe are columns - A,B,C and my target is Y Can my decision tree have a decision node which looks for say, ...
9
votes
1answer
3k views

When does decision tree perform better than the neural network?

I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree ...
99
votes
9answers
125k views

When should I use Gini Impurity as opposed to Information Gain (Entropy)?

Can someone practically explain the rationale behind Gini impurity vs Information gain (based on Entropy)? Which metric is better to use in different scenarios while using decision trees?
82
votes
6answers
124k views

strings as features in decision tree/random forest

I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, scikit-...
14
votes
4answers
58k views

How to make a decision tree with both continuous and categorical variables in the dataset?

Let's say I have 3 categorical and 2 continuous attributes in a dataset. How do I build a decision tree using these 5 variables? Edit: For categorical variables, it is easy to say that we will split ...
13
votes
2answers
30k views

How to normalize data for Neural Network and Decision Forest

I have a data set with 20000 samples, each has 12 different features. Each sample is either in category 0 or 1. I want to train a neural network and a decision forest to categorize the samples so that ...
8
votes
2answers
13k views

Multicollinearity in Decision Tree

Can anybody please explain the affect of multicollinearity on Decision Tree algorithms (Classification and regression). I have done some searching but was not able to find the right answer as some say ...
5
votes
2answers
3k views

Can a decision tree learn to solve a xOR problem?

I have read online that decision trees can solve xOR type problems, as shown in images (xOR problem: 1) and (Possible solution as decision tree: 2). My question is how can a decision tree learn to ...
11
votes
2answers
3k views

Is max_depth in scikit the equivalent of pruning in decision trees?

I was analyzing the classifier created using a decision tree. There is a tuning parameter called max_depth in scikit's decision tree. Is this equivalent of pruning a decision tree? If not, how could I ...
3
votes
1answer
84 views

Make a random forest estimator the exact same of a decision tree

The idea is to make one of the trees of a Random Forest, to be built exactly equal to a Decision Tree. First, we load all libraries, fit a decision tree and plot it. ...
0
votes
1answer
956 views

Decision tree not using all features from training dataset

I have built CART model using sklearn. I'm having total 6 features in training dataset and passing all of them in fit function. I've tested both criteria Gini and entropy. But whenever I plot tree ...
36
votes
3answers
35k views

Why do we need XGBoost and Random Forest?

I wasn't clear on couple of concepts: XGBoost converts weak learners to strong learners. What's the advantage of doing this ? Combining many weak learners instead of just using a single tree ? ...
29
votes
5answers
46k views

Are decision tree algorithms linear or nonlinear

Recently a friend of mine was asked whether decision tree algorithms are linear or nonlinear algorithms in an interview. I tried to look for answers to this question but couldn't find any satisfactory ...
7
votes
2answers
16k views

Why don't tree ensembles require one-hot-encoding?

I know that models such as random forest and boosted trees don't require one-hot encoding for predictor levels, but I don't really get why. If the tree is making a split in the feature space, then isn'...
19
votes
1answer
31k views

XGBRegressor vs. xgboost.train huge speed difference?

If I train my model using the following code: ...
11
votes
1answer
5k views

Why neural networks do not perform well on structured data?

I was recently working on some classification problem where decision trees performed better than neural networks. I had tried various combinations with neural networks altering the number of neurons / ...
12
votes
4answers
18k views

Interpreting Decision Tree in context of feature importances

I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the ...
10
votes
1answer
23k views

Decision tree, how to understand or calculate the probability/confidence of prediction result

For example, a drug prediction problem using a decision tree. I trained the decision tree model and would like to predict using new data. For example: ...
6
votes
1answer
6k views

Why we use information gain over accuracy as splitting criterion in decision tree?

In decision tree classifier most of the algorithms use Information gain as spiting criterion. We select the feature with maximum information gain to split on. I think that using accuracy instead of ...
12
votes
3answers
19k views

Unbalanced classes -- How to minimize false negatives?

I have a dataset that has a binary class attribute. There are 623 instances with class +1 (cancer positive) and 101,671 instances with class -1 (cancer negative). I've tried various algorithms (Naive ...
19
votes
2answers
15k views

Is it necessary to normalize data for XGBoost?

MinMaxScaler() in scikit-learn is used for data normalization (a.k.a feature scaling). Data normalization is not necessary for ...
9
votes
3answers
2k views

LightGBM - Why Exclusive Feature Bundling (EFB)?

I'm currently studying GBDT and started reading LightGBM's research paper. In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
3
votes
3answers
102 views

Better approach to assign values to determine potential fake sentences

I am trying to assign different values for each sentences based on information about the presence of hashtags, upper case letters/words (e.g. HATE) and some others. I created a data frame which ...
8
votes
1answer
151 views

Which ML approach to choose for the game AI when rewards are delayed?

Question: Which Machine Learning approach should I choose for the AI of my computer game, where the actions of the AI do not lead to immediate rewards, but delayed rewards instead? About me: I am a ...
8
votes
2answers
22k views

How to interpret a decision tree correctly?

I'm trying to work out if I'm correctly interpreting a decision tree found online. The dependent variable of this decision tree is Credit Rating which has two classes, Bad or Good. The root of this ...
3
votes
1answer
777 views

How is the 'feature_importance_' value calculated in sklearn random forest regressor?

I have 9000 sample, with five features, and one output variable (all are numerical, continuous values). I used random forest regression method using scikit modules. ...
11
votes
3answers
2k views

Can regression trees predict continuously?

Suppose I have a smooth function like $f(x, y) = x^2+y^2$. I have a training set $D \subsetneq \{((x, y), f(x,y)) | (x,y) \in \mathbb{R}^2\}$ and, of course, I don't know $f$ although I can evaluate $...
3
votes
1answer
3k views

selecting variable randomly at each node in a tree in Random Forest

In Random Forest method, for each tree we randomly select a set of variables (features) of fixed size. But once this set is frozen for that particular tree, does the tree behave like a regular ...
3
votes
1answer
3k views

Extract the "path" of a data point through a decision tree in sklearn

I'm working with decision trees in python's scikit learn. Unlike many use cases for this, I'm not so much interested in the accuracy of the classifier at this point so much as I am extracting the ...
1
vote
2answers
415 views

Can I force DecisionTreeClassifier to use integer conditions when the variable is integer?

I'm trying to visualize a decision tree in python for the purpose of explainability. I noticed that a condition like "NumGoals >= 1.23" could be quite vague for the user and I would much rather to see ...
5
votes
2answers
291 views

Sklearn: applying cost complexity pruning along with pipeline

I have a data set with categorical variables. I have defined a decision tree algorithm and transformed these columns to numerical equivalent using one hot encoding functionality in sklearn: Create ...
4
votes
2answers
393 views

Decision Trees Should We Discard Low Importance Features?

I just started to work with feature selection. Let's say I have a decision tree model. I get its feature importances by tree.feature_importances_. In my model out ...
4
votes
2answers
3k views

Fix first two levels of decision tree?

I am trying to build a regression tree with 70 attributes where the business team wants to fix the first two levels namely country and product type. To achieve this, I have two proposals: Build a ...
3
votes
2answers
6k views

Ordinal feature in decision tree

I am curious if ordinal features are treated differently from categorical features in decision tree, I am interested in both cases where target is categorical or continuous. If there is a difference, ...
2
votes
1answer
83 views

Can I create random forest with RandomForestClassifier which will consist the same trees?

Based on answers to this question, I should be able to build a random forest with all the same trees by using ...
2
votes
1answer
61 views

How does bagging help reduce the variance

I learned that bagging helps reduce variance by averaging but I couldn't understand this. Can someone explain this intuitively?
2
votes
4answers
3k views

is it possible to output more than 2 nodes away from a node in a decision tree? if yes, how to do that with sklearn?

usually a decision tree has one root node, some nodes, and some leaves. lots tutorial illustrate this as something like binary tree. is it possible more than 2 nodes away from a node in a decision ...
0
votes
1answer
303 views

How does class_weight work in Decision Tree?

I am interested in Cost-Sensitive learning. And I am trying to understand how class_weight in DecisionTree works in terms of math. I read a lot of articles that ...
9
votes
2answers
8k views

Why `max_features=n_features` does not make the Random Forest independent of number of trees?

Consider the following simple classification problem (Python, scikit-learn) ...
4
votes
1answer
145 views

Numeric variables in Decision trees

If we have numeric variable, decision trees will use < and > comparisons as splitting criteria. Lets consider this case : ...
3
votes
2answers
1k views

Ordinal Attributes in a Decision Tree

I'm reading the book Introduction to Data Mining by Tan, Steinbeck, and Kumar. In the chapter on Decision Trees, when talking about the "Methods for Expressing Attribute Test Conditions" the book says ...
2
votes
4answers
99 views

Predicting Disease Drugs

I have a dataset in the format: ...
2
votes
0answers
580 views

How to come up with the splitting point in a decision tree? [duplicate]

I read https://www.researchgate.net/post/How_to_compute_impurity_using_Gini_Index I understand why choosing smallest gini index, but how do I come up with different candidate splits in the first ...
1
vote
1answer
80 views

Decision Tree taking too long to execute

I am training a Decision Tree Regressor on a relatively small data. The dimensions of my train and test sets are (34164, 10) and (8514, 10). Here is the relevant code: ...
1
vote
1answer
323 views

XGBoost - Imputing Vs keeping NaN

What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
1
vote
1answer
623 views

decision -tree regression to avoid multicollinearity for regression model?

I read in comments a recommendation for decision tree´s instead of linear models like neural network, when the dataset has many correlated features. Because to avoid multicollinearity. A similar ...
1
vote
1answer
344 views

What can weka do that python and sklearn can't?

I would like to build a variety of classification and regression decision trees. My use case focuses on extraction and communication of decision rules. Previously weka was used in my organisation for ...