Skip to main content

Questions tagged [decision-trees]

A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm.

173 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8 votes
1 answer
4k views

Decision Trees - C4.5 vs CART - rule sets

When I read the scikit-learn user manual about Decision Trees, they mentioned that CART (Classification and Regression Trees) is very similar to C4.5, but it differs in that it supports numerical ...
emperorspride188's user avatar
6 votes
0 answers
12k views

Tuning Gradient Boosted Classifier's hyperparametrs and balancing it

I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated. Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model: ...
user1877600's user avatar
6 votes
1 answer
4k views

How does XGBoost compute the probabilities in predict_proba()?

I'm using the sklearn wrapper for XGBoost. I didn't manage to find a clear explanation for the way the probabilities given as output by predict_proba() are computed....
keren42's user avatar
  • 61
5 votes
0 answers
2k views

Dealing with categorical variables in Isolation Forest

Isolation Forest is widely used when dealing with outlier/anomaly detection when we have no labels. The theory behind is that making random split at random points and counting how many splits you do ...
Carlos Mougan's user avatar
4 votes
2 answers
743 views

Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax?

I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes ...
Caleb's user avatar
  • 151
4 votes
2 answers
4k views

Fix first two levels of decision tree?

I am trying to build a regression tree with 70 attributes where the business team wants to fix the first two levels namely country and product type. To achieve this, I have two proposals: Build a ...
Aravind's user avatar
  • 41
4 votes
1 answer
253 views

How important is lookahead search in decision trees?

I am using random forests, and, in my data, I have a lot of situations where $X_1$ is a bad predictor, $X_2$ is a bad predictor, but the joint distribution would make a good predictor. Say that $X1$, ...
Ricardo Cruz's user avatar
  • 3,420
3 votes
0 answers
180 views

Non-greedy decision tree / random forest implementation(s) in Python

The standard random forest is trained using a greedy approach for computational feasibility. However, there are a number of alternative methods such as "lookahead" or using bilevel ...
Peter's user avatar
  • 7,526
3 votes
0 answers
1k views

Combining heterogeneous numerical and text features

We want to solve a regression problem of the form "given two objects $x$ and $y$, predict their score (think about it as a similarity) $w(x,y)$". We have 2 types of features: For each ...
Dmitry's user avatar
  • 81
3 votes
1 answer
152 views

Output value of a gradient boosting decision tree node that has just a single example in it

The general gradient boosting algorithm for tree-based classifiers is as follows: Input: training set $\{(x_{i},y_{i})\}_{i=1}^{n}$, a differentiable loss function $L(y,F(x))$, and a number of ...
figs_and_nuts's user avatar
3 votes
3 answers
175 views

How to decide who to market? Clustering or Decision Tree?

I am working with a dataset that has enough observations and ~ 10 variables, half of the variables are numeric another half of the variables are categorical with 2-3 levels (demographics) one ID ...
Data Enthusiast's user avatar
3 votes
0 answers
40 views

Serializing a trained classification model into a set of actionable insights

I'm looking for ways to convert a trained classification model into a list of insights based on the resulting parameters of the model. To make an example, let's assume we trained a decision tree to ...
ozz1k's user avatar
  • 83
3 votes
0 answers
886 views

Target transformation for tree models

Can anybody explain why/if target variable transformations could help when dealing with tree based models? I've seen this excellent reply which explains quite well why it shouldn't affect if ...
Ludecan's user avatar
  • 261
3 votes
0 answers
54 views

Is linear regression on the trees of XGBoost (rather than taking their mean) useful/popular?

Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the ...
Andrew NC's user avatar
  • 131
3 votes
1 answer
537 views

Choosing a right algorithm for template-based text generation

I am doing a text generation project -- the task is to basically represent the statistical data in a readable way. The way I decided to go about this is template-based: each data type has a template ...
Vitaly Gorbachev's user avatar
3 votes
1 answer
270 views

Is decision tree regression comparable to locally weighted regression

I am new to decision tree method. For decision tree regression model, does it just fit a piece wise step function over data? When and why would people prefer it over some traditional regression like ...
ahala's user avatar
  • 131
3 votes
3 answers
264 views

CART algorithm (Classification and regression trees) question

We fit a full classification tree model $T_k$ of given depth $k$ to data using the CART algorithm, and prune the tree by finding $E(k, \alpha) = min_{T\subset Tk} Err(T) + \alpha |T|$. Here, $Err(T)$ ...
crommy's user avatar
  • 197
2 votes
0 answers
29 views

What are the analogies between decision trees and neural trees?

How can I draw analogies between decision trees and neural trees? For example, how are splitting thresholds analogous between these models, and how can paths in a neural tree be represented in a ...
Mir's user avatar
  • 71
2 votes
0 answers
20 views

Obtaining threshold based rules for classification problem

Suppose there are X1...Xn numerical variables predicting a target variable Y (0 or 1) Objective: to obtain the best possible thresholds and combinations of X1...Xn that can predict Y Example: (X1>...
Sunit Gautam's user avatar
2 votes
1 answer
51 views

Classifying short strings of text with additional context

I have a list of short strings each identifying a city. Misspellings are very common. The example below shows some of these short strings, along with the correct city they're supposed to match. ...
Jivan's user avatar
  • 165
2 votes
0 answers
87 views

How to implement an oblique decision tree for regression?

There are numerous ways to induce an oblique decision tree in the decision tree induction domain, such as using a support vector machine to determine the best hyper-plane. However, is it possible to ...
HZ-VUW's user avatar
  • 121
2 votes
2 answers
1k views

How to decode encoded labels in Decision tree classifier

I have some dataset with procurements of organization where actually i'm working. The aim is to find most important features that describe why some processes of purchases is succesful, and why not ...
O.Sartaev's user avatar
2 votes
1 answer
33 views

Which models implicitly consider interaction between features?

I would like to understand more how different models (NN and RF specifically, but any other as well) consider interaction between features in tabular data? For example, can the model figure out while ...
xxanissrxx's user avatar
2 votes
2 answers
617 views

Decision Tree Induction using Information Gain and Entropy

I’m trying to build a decision tree algorithm, but I think I misinterpreted how information gain works. Let’s say we have a balanced classification problem. So, the initial entropy should equal 1. ...
Krushe's user avatar
  • 21
2 votes
0 answers
466 views

Aggregate SHAP importances from different models

A couple of questions on the SHAP approach to the estimation of feature importance. I would like to use the random forest, logistic regression, SVM, and kNN to train four classification models on a ...
CaffeineMan's user avatar
2 votes
1 answer
2k views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using the h2o library for Python to build a decision tree and to extract the decision rules out of it. I am using some data for training where ...
Sapiens's user avatar
  • 21
2 votes
0 answers
402 views

why does ID3 Decision tree algorithm not give the best decision tree?

I was going through ID3 algorithm, and what I believe is it incorporates Greedy Search rule to get come up with the decision tree. If it gives the best split possible at every stage, how does it not ...
MrRobot9's user avatar
  • 149
2 votes
0 answers
44 views

How to turn classification tree results into a GIS map

I'm new-ish to machine learning, so this could be a silly question. Apologies if so. The idea is is to predict groundwater occurrence based on a regression tree. This is my conceptual model: Target ...
jerome's user avatar
  • 21
2 votes
1 answer
227 views

Is it feasible to use decision tree algorithms for sensor fault detection?

The gist is me wanting to separate system faults from sensor faults given some dataset from a wireless sensor network using a machine learning algorithm. For instance, if I have some temperature ...
Aldazar's user avatar
  • 21
2 votes
0 answers
587 views

Sales Dataset to determine best model for predicting future sales

We have a set of products in which we are trying to determine which products we should continue to sell, and which products to remove from our inventory. The file contains BOTH historical sales data ...
David B's user avatar
  • 29
2 votes
0 answers
153 views

Parameters for CART tree

I'm using fitctree function from Matlab to fit a CART tree to my dataset (10'000 data points, 20 features). First, how can I exactly prune it and to what level should the tree be pruned? I think I ...
machinery's user avatar
  • 246
2 votes
1 answer
98 views

Decision tree to get difference in rates in two groups?

I have two sample groups of customers, each customer has 100s of features. For a single sample, i would use Decision Trees to find sub-groups that have a high churn rate. Thats easy. However, my ...
Arslán's user avatar
  • 131
2 votes
1 answer
93 views

Machine Learning - Same impurity values

Imagine this example: We see that the attributes color and points have the same value. What attribute should we choose for the ...
Juan Ramos's user avatar
1 vote
1 answer
28 views

Neural Network Weights - How do they know their position?

I am a copyright scholar so please forgive my ignorance. When weights are stored external to a model what is the mechanism by which the weight knows which neuron or node in a decision tree it is ...
Benjamin White's user avatar
1 vote
0 answers
30 views

How to know which rules were applied to predict one sample in trained decision tree model?

I have trained Random Forest Regressor from sklearn. I am able to return text representation from each Decision Tree rule using tree.export_text (sklearn documentation here). But it shows rules for ...
Paulina's user avatar
  • 15
1 vote
0 answers
47 views

One-Hot encoded variables dominates importance among other variables

I am currently training some machine learning models to predict the 28-day compressive strength of cement, a continuous real-valued variable. The available dataset comprises samples from three ...
Felipe's user avatar
  • 21
1 vote
1 answer
24 views

Regression models that conform to functional groupings of features

For example, suppose we want to predict y with features x1, x2, x3, x4. If I specify ...
olives's user avatar
  • 11
1 vote
1 answer
32 views

XGBoost prints trees beyond n_estimator

I have a XGBoost model with the following parameters ...
Itminan's user avatar
  • 11
1 vote
1 answer
29 views

How to use labels to fit several thresholds in a simple decision rule?

I have a binary labelled dataset with numeric features. I want to create a "business rule" of the type y = x1 > t1 and x2 > t2 and x3 > t3. ...
hipoglucido's user avatar
  • 1,170
1 vote
0 answers
12 views

Precision and AUROC for which class values

I am a newbie in reading research paper and implementing it by myself. I went through the paper Breast Cancer Survival Prediction from Imbalanced Dataset with Machine Learning Algorithms. Can anyone ...
Encipher's user avatar
  • 361
1 vote
0 answers
31 views

ML Model that doesnt average/penalize extreme values

I have a 20k dataset, and a couple hundred of those lines are extreme values and 10 of them or so are even extremer values. But they are correct and have a unique tag, so when that tag comes up I am ...
Jroc561's user avatar
  • 11
1 vote
0 answers
108 views

Error from XGBoost missing data handling

I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
lexan55's user avatar
  • 36
1 vote
0 answers
52 views

Machine learning frameworks for tree-based models

Background: Its well known that Pytorch and TensorFlow are currently the most used frameworks for Deep Learning (DL) research. As far as I know, most researchers (applied or theoretical) that ...
Enk9456's user avatar
  • 105
1 vote
0 answers
18 views

Regression tree cross validation confusing results

Setup I have implemented regression trees in go: full repository Using the full dataset and cost-complexity pruning, I get the following alphas and corresponding average residual (again, against the ...
wheresmycookie's user avatar
1 vote
1 answer
238 views

Understanding feature_parallel distributed learning algorithm in LightGBMClassifier

I want to understand feature_parallel algorithm in LightGBMClassifier. It describes how it is done traditionally and how LightGBM...
figs_and_nuts's user avatar
1 vote
1 answer
416 views

SKLearn decisionTreeClassifier does not handle sparse or categorical data

Is there a way in fitting a decisionTreeClassifier in SKLearn to sparse tuples? The data that I have is based on about 100 features, but only a few of them are ever used to make the decision. ...
Bruce's user avatar
  • 196
1 vote
0 answers
763 views

Theoretical maximum depth of a decision tree

During my machine learning labwork, I was trying to fit a decision tree to the IRIS dataset (150 samples, 4 features). The maximum theoretical depth my tree can reach which is, for my understanding, ...
Souhaielrmx's user avatar
1 vote
0 answers
28 views

Difference between rpart models, one with information split the other with rpart.control

What is the difference between these two models? ...
cocoakrispies98's user avatar
1 vote
0 answers
18 views

What kind of model to use to find drivers when data is aggregated and not on user level?

I have a website and have info from Google Analytics. So I can see the following "features": page url country device category (phone, desktop, etc.) Number of sessions Number of users: ...
user126224's user avatar
1 vote
0 answers
42 views

How are regression trees fitted in gradient boosting for classification?

What I understood is that even gradient boosting for binary classification uses regression trees. The first value we calculate is constant = log(odds). For the rest of the trees, we try to fit ...
Nikhil Mishra's user avatar