Questions tagged [boosting]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
18 views

Tree-based algorithms and ordinal features

For tree-based methods (e.g., DT, Random Forest, Gradient boosting, etc.), does the conversion interval of an ordinal feature to continuous matter matters? (I can see why it matters for linear model, ...
1
vote
1answer
20 views

Counting the number of trainable parameters in a gradient boosted tree

I recently ran the gradient boosted tree regressor using scikit-learn via: GradientBoostingRegressor() This model depends on the following hyperparameters: Estimators ($N_1$) Min Samples Leaf ($N_2$...
2
votes
0answers
21 views

How to improve model performace when model shows a systemic pattern in residues

I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
5
votes
1answer
23 views

Is the way to combine weak learners in AdaBoost for regression arbitrary?

I'm reading about how variants of boosting combine weak learners into final predication. The case I'm consider is regression. In paper Improving Regressors using Boosting Techniques, the final ...
0
votes
0answers
18 views

Understanding interplay between eval_metric, metric and first_metric_only parameters in LGBMClassifier

In python API of LGBMClassifier, the constructor takes parameters metric and first_metric_only. Their descriptions are as ...
3
votes
1answer
21 views

What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another?

I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding: hessian ...
0
votes
0answers
9 views

A question about SAMME algorithm (Adaboost)

According to this article (which references the original paper), this is the SAMME algorithm for multiclass classification using Adaboost: I would like to understand what is this term in step ...
2
votes
0answers
20 views

What is the concept behind the categorical-encoding used in the CatBoost benchmark problems?

I'm working through CatBoost quality benchmark problems (here). I'm particularly intrigued by the methodology adopted to convert categorical features to numerical values as described in the ...
0
votes
0answers
24 views

Solving the dual problem of boosting using column generation

In our book there is boosting algorithm using column generation method (Dantzig-Wolfe decomposition) to solve the dual problem. So lets say we have want to solve the following primal linear problem ...
1
vote
2answers
73 views

Does gradient boosting algorithm error always decrease faster and lower on training data?

I am building another XGBoost model and I'm really trying not to overfit the data. I split my data into train and test set and fit the model with early stopping based on the test-set error which ...
0
votes
0answers
20 views

Why do you need to update the number of boosting rounds each time you update a parameter in xgboost?

I have been reading material that suggesting that, after each grid search you do on a single parameter (e.g say on learning rate), you should update the number of boosting rounds afterwards. This is ...
1
vote
1answer
55 views

splitting mechanism with one hot encoded variables (tree based/boosting)

I am using xgboost and have a categorical unordered feature with 25 levels. So when i apply one hot encoding i have 25 columns. This introduces alot of sparsity. Even more unusual, my feature ...
0
votes
0answers
39 views

AdaBoost.R2 learning rate from scikit learn

AdaBoost.R2 (regression), is presented in the paper "improving regressors with boosting techniques" from Drucker and is freely available on Scholar. The implementation of regression for ...
0
votes
0answers
36 views

How does reweighting samples actually affect each new weak learner in adaboost?

In Adaboost, when you reweight the samples, how does the training process for the next weak learner in the boosting algorithm take in to account the weights? Is it reflected in the loss function of ...
0
votes
1answer
44 views

Which other algorithms fit residuals like XGBoost? [closed]

XGBoost and standard gradient boosting train learners to fit the residuals rather than the observations themselves. I understand that this aspect of the algorithm matches the boosting mechanism which ...
0
votes
0answers
37 views

Getting unexpected keyword error in CatBoostRegressor while using GridSearchCV

I am trying to use GridSearchCV on a CatBoostRegressor algorithm, but get some "unexpected keyword" errors on 3 different params (classes_count, auto_class_weights, and bayesian_matrix_reg) ...
3
votes
1answer
47 views

On gradient boosting and types of encodings

I am having a look at this material and I have found the following statement: For this class of models [Gradient Boosting Machine algorithms] [...] it is both safe and significantly more ...
1
vote
1answer
13 views

Explanation on some steps of AdaBoost.R2

I am trying to understand AdaBoost.R2 in order to implement it and apply it to a regression problem. In this circumstances I need to understand it perfectly, however there's some step i don't really ...
2
votes
0answers
15 views

Understanding additive function approximation or Understanding matching pursuit

I am trying to read Greedy function approximation: A gradient boosting machine. On page 4 (it is marked as page 1192) under 3. Finite data the author tells how the function approximation approach ...
4
votes
3answers
254 views

Can Boosted Trees predict below the minimum value of the training label?

I am using gradient Gradient Boosted Trees (with Catboost) for a Regression task. Can GBtrees predict a label that is below the minimum (or above the max) that was seen in the training ? For instance ...
1
vote
1answer
38 views

is it possible get a overfit underfit comparation between models, with this chart? (homework) [closed]

I am trying to interpret this chart. I am not sure how to interpret this, because, I think that the fact of the for examples LGBM Validation error, is wide and similar to train boxplot, there arent ...
10
votes
2answers
743 views

What is a good interpretation of this 'learning curve' plot?

I read about the validation_curve and how interpret it to know if there are over-fitting or underfitting, but how can interpret the plot when the data is the error like this: The X-axis is "Nº ...
0
votes
0answers
44 views

Confidence score over xgboost logistic regression

The probabilities of logistic regression indicate how the certain the model is over predictions. if its 0.93 it means the model is 93% confident the label is 1 and 7% to be 0. or if the probability is ...
0
votes
0answers
14 views

RMSE in NGBoost example appears to be unreasonably high

I tried using Standford's ML Group NGBoost. I tried using this example provided by the authors of the repository. While training the output is along the lines of : ...
0
votes
0answers
31 views

XGBOOST/lLightgbm over-fitting despite no indication in cross-validation test scores?

We currently work on a project where we aim to identify a set of predictors that may influence the risk of a relatively rare outcome. We are using a semi-large clinical dataset, with data on nearly ...
0
votes
0answers
13 views

Building Uplift Tree using boosting

I want to build an Uplift Model for multiple treatments. To get a good model, I would like to use boosting. How is it possible to use boosting with uplift modeling although we can't really calculate ...
1
vote
0answers
10 views

Can i use other regression types that arent based in decision trees to use it like a weak learners in gradient boosting?

I was thinking if i can use polynomial regression like a weak learners in gradient boosting but i read that decision trees are used for that and i cannot find anything that show me the possibility of ...
1
vote
0answers
17 views

Tuning parameters for gradient boosting/xgboost

In practice, which parameter do you typically tune first? Do you tune the learning rate (or step size) first? and then tune the total number of iterations? And how do you go about tuning these ...
2
votes
1answer
35 views

Random Forest Stacking Experiment for Imbalanced Data-set Problem

In order to solve a Imbalanced Dataset Problem, I experimented with Random Forest in the given manner (Somewhat inspired by Deep-Learning) Trained a Random Forest which will take in the input data ...
6
votes
1answer
315 views

Boosting with highly correlated features

I have a conceptual question. My understanding is, that Random Forest can be applied even when features are (highly) correlated. This is because with bagging, the influence of few highly correlated ...
2
votes
1answer
95 views

Decreasing n_estimators is increasing accuracy in AdaBoost?

I was exploring the AdaBoost classifier in sklearn. This is the plot of the dataset. (X,Y are the predictor columns and the color is the label) As you can see there are exactly 16 points in either ...
1
vote
0answers
20 views

Do you need to perform variables reduction for tree-based models?

I know for methods and linear regression, GLM, Logistic regression, we typically run through a lot of variable reduction methods, i.e, forward/backward/stepwise, univariate analysis; variable ...
2
votes
1answer
111 views

Extracting encoded features after CatBoost

I have a dataset containing numerical as well as categorical variables. After I've fit my dataset to a CatBoostClassifier, I want to extract the entire feature set, with the categorical variables ...
0
votes
1answer
114 views

Getting feature vectors from CatBoost pool

I have a dataset with some numerical and categorical features and I am trying to apply CatBoost for categorical encoding and classification. Since my dataset is highly imbalanced, with a large number ...
0
votes
1answer
151 views

Handling Categorical Features on NGBoost

Recently I have been doing some research on NGBoost, but I could not see any parameter for categorical features. Is there any parameter that I missed? ...
4
votes
5answers
2k views

GridSearch without CV

I create a Random Forest and Gradient Boosting Regressor by using GridSearchCV. For the Gradient Boosting Regressor it takes too long for me. But i need to know which are the best Parameter for the ...
2
votes
1answer
111 views

Extremely high gain with LightGBM

I am working on a binary classification problem. The target variable is not linearly separable, so I've decided to use LightGBM with default parameters (I only play with n_estimators on range from 10 -...
1
vote
1answer
32 views

Time series with additional information

Given a time series with job-submission counts, how can I predict which certain features about the jobs? I need to predict how many jobs and which jobs arrived in some system. Using ...
3
votes
0answers
84 views

weighted quantile sketch in xgboost

I am unable to understand what is weighted quantile sketch in xgboost. Can anyone help me give an intuitive understanding of this?
5
votes
1answer
372 views

What is Pruning & Truncation in Decision Trees?

Pruning & Truncation As per my understanding Truncation: Stop the tree while it is still growing so that it may not end up with leaves containing very low data points. One way to do this is to set ...
1
vote
1answer
281 views

best way to regularize gradient boosting regressor?

i am testing gradient boosting regressor from sklearn for time series prediction on noisy data (currency markets). https://scikit-learn.org/stable/modules/generated/sklearn.ensemble....
0
votes
0answers
13 views

Improve the results of imbalanced multi-classification multi-lables data

I have 10k rows of multi-classification (x1..x27,y), size of dataframe is: 28*10k and its ...
5
votes
2answers
323 views

Bagging vs Boosting, Bias vs Variance, Depth of trees

I understand the main principle of bagging and boosting for classification and regression trees. My doubts are about the optimization of the hyperparameters, especially the depth of the trees First ...
1
vote
1answer
211 views

Error rate of AdaBoost weak learner always bigger than 0.5?

As far as i understand, weak learners of AdaBoost should never yield a error rate > 0.5 After training one, i only receive error rates above 0.5. How is that even possible? The AdaBoost Tree still ...
0
votes
1answer
23 views

Ensemble Techniques - Boosting

I understand boosting is a sequential learning technique and it use the prediction from previous model as a dataset for new model ,after adding weight to the misclassified data points. The point ...
0
votes
1answer
46 views

Similarity of XGBoost models?

Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?
3
votes
2answers
524 views

XGBoost validation for number of trees

I have a simple Question: I am using XGBoost to classify some data: 1.) With 100 estimators I have following scores(roc_score): train_data : 98.5 validation_data : 97.2 2.) With 500 ...
0
votes
1answer
24 views

How to tell a boosting model that 2 features are related and should not be interpreted stand-alone?

I am using XGBoost for a machine learning model that learns from tabular data. XGBoost uses boosting method on decision trees. When I look at the decision-making logic of decision trees, I notice the ...
1
vote
1answer
30 views

How to control the amount of positives in classification?

I have a basic, yet quite complex problem to solve right now. Let's say we have a training set of 20,000 samples in my training set, out of which 3 to 4% is flagged as "True", the rest is flagged as "...
1
vote
0answers
25 views

Purpose of gamma multiplier in gradient boosting

looking through the mathematics of gradient boosting on the relevant wikipedia page, intuitively what is the purpose of the multiplier $\gamma_i$? This term does not appear in the following ...