Questions tagged [decision-trees]

A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm.

184 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
145 views

Multi-target regression tree with additional constraint

I have a regression problem where I need to predict three dependent variables ($y$) based on a set of independent variables ($x$):  (y_1,y_2,y_3) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \...
• 6,872
3k views

Decision Trees - C4.5 vs CART - rule sets

When I read the scikit-learn user manual about Decision Trees, they mentioned that CART (Classification and Regression Trees) is very similar to C4.5, but it differs in that it supports numerical ...
974 views

How does XGBoost compute the probabilities in predict_proba()?

I'm using the sklearn wrapper for XGBoost. I didn't manage to find a clear explanation for the way the probabilities given as output by predict_proba() are computed. In random forest for example, I ...
• 51
11k views

Tuning Gradient Boosted Classifier's hyperparametrs and balancing it

I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated. Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model: ...
• 276
101 views

How is bayesian risk computed to prune decision trees?

I've been trying to follow this paper on Bayesian Risk Pruning. I'm not very familiar with this type of pruning, but I'm wondering a few things: (1) The paper describes risk-rates to be defined per ...
• 41
1k views

Dealing with categorical variables in Isolation Forest

Isolation Forest is widely used when dealing with outlier/anomaly detection when we have no labels. The theory behind is that making random split at random points and counting how many splits you do ...
• 5,436
1k views

how does XGBoost's exact greedy split finding algorithm determine candidate split values for different feature types?

Based on the paper by Chen & Guestrin (2016) "XGBoost: A Scalable Tree Boosting System", XGBoost's "exact split finding algorithm enumerates over all the possible splits on all the features to ...
• 51
3k views

Fix first two levels of decision tree?

I am trying to build a regression tree with 70 attributes where the business team wants to fix the first two levels namely country and product type. To achieve this, I have two proposals: Build a ...
• 41
193 views

How important is lookahead search in decision trees?

I am using random forests, and, in my data, I have a lot of situations where $X_1$ is a bad predictor, $X_2$ is a bad predictor, but the joint distribution would make a good predictor. Say that $X1$, ...
74 views

Non-greedy decision tree / random forest implementation(s) in Python

The standard random forest is trained using a greedy approach for computational feasibility. However, there are a number of alternative methods such as "lookahead" or using bilevel ...
• 6,872
54 views

Output value of a gradient boosting decision tree node that has just a single example in it

The general gradient boosting algorithm for tree-based classifiers is as follows: Input: training set $\{(x_{i},y_{i})\}_{i=1}^{n}$, a differentiable loss function $L(y,F(x))$, and a number of ...
71 views

How to decide who to market? Clustering or Decision Tree?

I am working with a dataset that has enough observations and ~ 10 variables, half of the variables are numeric another half of the variables are categorical with 2-3 levels (demographics) one ID ...
30 views

Serializing a trained classification model into a set of actionable insights

I'm looking for ways to convert a trained classification model into a list of insights based on the resulting parameters of the model. To make an example, let's assume we trained a decision tree to ...
• 63
658 views

Target transformation for tree models

Can anybody explain why/if target variable transformations could help when dealing with tree based models? I've seen this excellent reply which explains quite well why it shouldn't affect if ...
• 221
52 views

Is linear regression on the trees of XGBoost (rather than taking their mean) useful/popular?

Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the ...
• 131
245 views

Choosing a right algorithm for template-based text generation

I am doing a text generation project -- the task is to basically represent the statistical data in a readable way. The way I decided to go about this is template-based: each data type has a template ...
185 views

Is decision tree regression comparable to locally weighted regression

I am new to decision tree method. For decision tree regression model, does it just fit a piece wise step function over data? When and why would people prefer it over some traditional regression like ...
• 131
560 views

Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax?

I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes ...
• 141
245 views

Does CART algorithm takes into account in the order of the set of attributes?

when using matlab command 'fitctree' for classification purpose, and I change the order of the attributes I do not find the same Tree and thus the same classificaiton error? why? CART algorithm does ...
• 43
192 views

CART algorithm (Classification and regression trees) question

We fit a full classification tree model $T_k$ of given depth $k$ to data using the CART algorithm, and prune the tree by finding $E(k, \alpha) = min_{T\subset Tk} Err(T) + \alpha |T|$. Here, $Err(T)$ ...
• 197
12 views

Obtaining threshold based rules for classification problem

Suppose there are X1...Xn numerical variables predicting a target variable Y (0 or 1) Objective: to obtain the best possible thresholds and combinations of X1...Xn that can predict Y Example: (X1>...
• 121
22 views

Classifying short strings of text with additional context

I have a list of short strings each identifying a city. Misspellings are very common. The example below shows some of these short strings, along with the correct city they're supposed to match. ...
• 165
879 views

Combining heterogeneous numerical and text features

We want to solve a regression problem of the form "given two objects $x$ and $y$, predict their score (think about it as a similarity) $w(x,y)$". We have 2 types of features: For each ...
• 71
33 views

How to implement an oblique decision tree for regression?

There are numerous ways to induce an oblique decision tree in the decision tree induction domain, such as using a support vector machine to determine the best hyper-plane. However, is it possible to ...
• 121
469 views

How to decode encoded labels in Decision tree classifier

I have some dataset with procurements of organization where actually i'm working. The aim is to find most important features that describe why some processes of purchases is succesful, and why not ...
22 views

Which models implicitly consider interaction between features?

I would like to understand more how different models (NN and RF specifically, but any other as well) consider interaction between features in tabular data? For example, can the model figure out while ...
• 103
369 views

Decision Tree Induction using Information Gain and Entropy

I’m trying to build a decision tree algorithm, but I think I misinterpreted how information gain works. Let’s say we have a balanced classification problem. So, the initial entropy should equal 1. ...
• 21
157 views

Use cross entropy to create decision tree classifier

Are entropy and cross-entropy the same thing as per basic definition? If there is a difference: Decision tree splits take on entropy or Gini index, can we use cross-entropy to split decision trees? ...
• 151
1k views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using the h2o library for Python to build a decision tree and to extract the decision rules out of it. I am using some data for training where ...
• 21
31 views

How to turn classification tree results into a GIS map

I'm new-ish to machine learning, so this could be a silly question. Apologies if so. The idea is is to predict groundwater occurrence based on a regression tree. This is my conceptual model: Target ...
• 21
181 views

Is it feasible to use decision tree algorithms for sensor fault detection?

The gist is me wanting to separate system faults from sensor faults given some dataset from a wireless sensor network using a machine learning algorithm. For instance, if I have some temperature ...
• 21
572 views

Sales Dataset to determine best model for predicting future sales

We have a set of products in which we are trying to determine which products we should continue to sell, and which products to remove from our inventory. The file contains BOTH historical sales data ...
• 29
141 views

Parameters for CART tree

I'm using fitctree function from Matlab to fit a CART tree to my dataset (10'000 data points, 20 features). First, how can I exactly prune it and to what level should the tree be pruned? I think I ...
• 234
85 views

Decision Trees and Categorical Feature Labelling

I am working on a decision tree model and trying to decide how best to handle categorical features. The features in my dataset are generally high in cardinality and I have found that ordinal labeling ...
• 21
61 views

Decision tree to get difference in rates in two groups?

I have two sample groups of customers, each customer has 100s of features. For a single sample, i would use Decision Trees to find sub-groups that have a high churn rate. Thats easy. However, my ...
• 131
79 views

Machine Learning - Same impurity values

Imagine this example: We see that the attributes color and points have the same value. What attribute should we choose for the ...
• 121
1 vote
19 views

Error from XGBoost missing data handling

I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
• 21
1 vote
14 views

Machine learning frameworks for tree-based models

Background: Its well known that Pytorch and TensorFlow are currently the most used frameworks for Deep Learning (DL) research. As far as I know, most researchers (applied or theoretical) that ...
• 89
1 vote
15 views

Regression tree cross validation confusing results

Setup I have implemented regression trees in go: full repository Using the full dataset and cost-complexity pruning, I get the following alphas and corresponding average residual (again, against the ...
1 vote
48 views

Understanding feature_parallel distributed learning algorithm in LightGBMClassifier

I want to understand feature_parallel algorithm in LightGBMClassifier. It describes how it is done traditionally and how LightGBM...
1 vote
203 views

Theoretical maximum depth of a decision tree

During my machine learning labwork, I was trying to fit a decision tree to the IRIS dataset (150 samples, 4 features). The maximum theoretical depth my tree can reach which is, for my understanding, ...
1 vote
15 views

Difference between rpart models, one with information split the other with rpart.control

What is the difference between these two models? ...
1 vote
13 views

What kind of model to use to find drivers when data is aggregated and not on user level?

I have a website and have info from Google Analytics. So I can see the following "features": page url country device category (phone, desktop, etc.) Number of sessions Number of users: ...
1 vote
73 views

How do I know if this model is overfitting?

This is my example R script for a decision tree: ...
• 459
1 vote
21 views

How are regression trees fitted in gradient boosting for classification?

What I understood is that even gradient boosting for binary classification uses regression trees. The first value we calculate is constant = log(odds). For the rest of the trees, we try to fit ...
1 vote
30 views

Classification for Ordinal labels - what tree-based methds can i use?

I have a label that has a natural ordering e.g. 0,1,2,3 where 0 is the worst activity measure and 3 is the best. For each label given by the model i need to also give the probability that it belongs ...
• 466
1 vote
83 views

Is it possible to do hard-coded decision tree on some variables and random forest / something on the remaining ones?

Is it possible to do hard-coded decision tree on some variables and random forest / something on the remaining ones? The situation seems that for some variables it's possible to draw strong empirical ...
• 379
1 vote
10 views

Does anyone know of literature regarding a Neural Net boosted GBM?

For obvious reasons, most GBMs created in the private sector are tree boosted. Occasionally, one might want a linear boosted GBM so that the residual models collapse into a simple linear combination. ...
1 vote