People who code: we want your input. Take the Survey

Questions tagged [ensemble-modeling]

In machine learning, ensemble methods combine multiple algorithms to make a prediction. Bagging, boosting, and stacking, are some examples.

Filter by
Sorted by
Tagged with
59
votes
5answers
83k views

GBM vs XGBOOST? Key differences?

I am trying to understand the key differences between GBM and XGBOOST. I tried to google it, but could not find any good answers explaining the differences between the two algorithms and why xgboost ...
20
votes
1answer
25k views

Adaboost vs Gradient Boosting

How is AdaBoost different from a Gradient Boosting algorithm since both of them use a Boosting technique? I could not figure out actual difference between these both algorithms from a theory point of ...
8
votes
1answer
4k views

what is the difference between “fully developed decision trees” and “shallow decision trees”?

As reading Ensemble methods on scikit-learn docs, it says that bagging methods work best with strong and complex models (e.g., fully developed decision trees), in contrast with boosting methods ...
7
votes
5answers
2k views

Does ensemble (bagging, boosting, stacking, etc) always at least increase performance?

Ensembling is getting more and more popular. I understand that there are in general three big fields of ensembling, bagging, boosting and stacking. My question is that does the ensembling always at ...
7
votes
2answers
212 views

Why isn't dimension sampling used with gradient boosting machines (GBM)?

GBMs, like random forests, build each tree on a different sample of the dataset and hence, going by the spirit of ensemble models, produce higher accuracies. However, I have not seen GBM being used ...
6
votes
3answers
240 views

What are the individual models within a machine learning ensemble called?

I am aware that an ensemble machine learning model is a stack of two or more machine learning models. Is there a word to refer to those individual models that go into the ensemble model? (i.e. a ...
6
votes
1answer
722 views

Is there any difference between a weak learner and a weak classifier?

While reading about decision tree ensembles Gradient Boosting, AdaBoost etc. I have found the following two concepts weak learner and weak classifier. Are they the same? If there is any difference ...
6
votes
1answer
218 views

How predictions of level 1 models become training set of a new model in stacked generalization.

In stacked generalization, if I understood well, we divide the training set into train/test set. We use train set to train M models, and make predictions on test set. Then we use the predictions as ...
6
votes
1answer
9k views

Assumptions/Limitations of Random Forest Models

What are the general assumptions of a Random Forest Model? I could not find by searching online. For example, in a linear regression model, limitations/assumptions are: It may not work well when ...
5
votes
2answers
192 views

Stacking and Ensembling methods in Data Science

I understand that using stacking and ensembling has become popular, and these methods can give better results than using a single algorithm. My question is: What are the reasons, statistical or ...
5
votes
1answer
169 views

2 stage ensemble — CV MSE valid in 1st stage but not in 2nd

I'm trying out a Kaggle competition, which puts me in the unusual position of being able to get feedback on my models' "true" performance (you can submit several predictions per day and they give you ...
5
votes
2answers
27k views

What does Negative Log Likelihood mean?

I have a data set which has continuous independent variables and a continuous dependent variable. To predict the dependent variable using the independent variables, I've run an ensemble of regression ...
5
votes
3answers
256 views

How to create an ensemble that gives precedence to a specific classifier

Suppose that in a binary classification task, I have separate classifiers A, B, and C. If I ...
4
votes
3answers
2k views

Ensembling vs clustering in machine learning

Following the raise of ensembling (e.g ensembling of xgboost learners) after its recurrent wins in Kaggle competitions, using it has become a tradition in machine learning. However, some argue that ...
4
votes
1answer
3k views

Sklearn Aggregating Multiple Fitted Models Into A Single Model? (binary classification)

My problem context: dataset too big to fit into memory. binary classification [0,1] 30 csv files in a directory with exactly 30,000 samples (rows) each file contains 15,000 ...
4
votes
2answers
662 views

What is the meaning of the term “pipeline” within data science?

Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here People often refer to pipelines when talking about models, data and even layers ...
4
votes
2answers
1k views

SHAP value can explain right?

I face a problem with using SHAP value to interpret the Tree-based model. (https://github.com/slundberg/shapsd) First, I have input around 30 features and I have 2 features that have high positive ...
4
votes
2answers
3k views

Categorical data for sklearns Isolation Forrest

I'm trying to do anomaly detection with Isolation Forests (IF) in sklearn. Except for the fact that it is a great method of anomaly detection, I also want to use it because about half of my features ...
4
votes
1answer
46 views

Classification model accuracy with ensemble methods

I came through this statement in a Machine Learning text book based on law of large numbers: Suppose you build an ensemble containing 1,000 classifiers that are individually correct only 51% of the ...
4
votes
1answer
928 views

Geometric and harmonic means in ensembling methods

When using ensembling methods for regression, a common approach is to average (using the arithmetic mean) the outputs of the weak learners in order to obtain the output of the ensemble. Is there a ...
4
votes
2answers
1k views

How to ensemble classifier incorporating all features in python?

I am doing a text classification task(5000 essays evenly distributed by 10 labels). I explored LinearSVC and got an accuracy of 80%. Now I guess whether accuracy ...
4
votes
2answers
139 views

clustering plus linear model versus non linear (tree) model

a team has to create models that predict the cost of deploying a machine over time. This is a regression problem. The team is further divided into two groups, A and B. Group A puts lots of ...
4
votes
0answers
2k views

How to tune weights in Voting Classifier (Sklearn)

I am trying to do the following: ...
3
votes
2answers
198 views

Probability that ensemble model is correct based on accuracies of its classifiers

I'm trying to understand what I did wrong when trying to answer this question. The exact question is: Assume that we have 3 trained prediction models, and each model outputs either -1 or 1. We then ...
3
votes
1answer
488 views

Is SuperLearning actually different to stacking,or are they essentially the same thing?

Articles which use the terms 'stacking' and 'Super Learner' often seem to use the terms interchangeably. Is the Super Learner algorithm a specific form of the more generic stacking concept, or is ...
3
votes
2answers
1k views

Improving classifier performances in R for imbalanced dataset

I have used an "adabag"(boosting + bagging) model on an imbalanced dataset (6% positive), I have tried to maximized the sensitivity while keeping the accuracy above 70% and the best results I got were:...
3
votes
1answer
198 views

Are “Gradient Boosting Machines (GBM)” and GBDT exactly the same thing?

In the category of Gradient Boosting, I find some terms confusing. I'm aware that XGBoost includes some optimization in comparison to conventional Gradient Boosting. But are Gradient Boosting ...
3
votes
1answer
3k views

Is there any implementation of Extended Isolation Forest algorithm in R/Python?

I am using isofor package for regular Isolation Forest but I came by an article about Extended Isolation Forest and need your advise which package has this ...
3
votes
1answer
46 views

Building Stacking machine learning model using three base classifiers

I did a stacking using three base classifiers RF, NB, KN N and metamodel random forest or SVM using sklearn library But which is strange each time i change the metamodel i got the same results. Is it ...
3
votes
1answer
94 views

Why some people add results from PCA and other dimensional reductions techniques as features

I often see in a well-known datascience competition platform, that a lot of people apply some dimension reductions techniques, but instead of using it to reduce the number of features (complexity) of ...
3
votes
1answer
180 views

Weighted Linear Combination of Classifiers

I am trying to build an ensemble of classifiers whereby I want my algorithm to learn a set of weights such that it can weight the outputs of different classifiers for a set of data points. I am ...
3
votes
1answer
215 views

Ensemble Techniques for multilabel data

I observed that Adaboost or Bagging ensemble classifiers present in sklearn only work for single label training data. How do I use these for multilabel data?
3
votes
1answer
2k views

Algorithmic approach to model blending

Model blending -- by which I mean creating multiple sets of predictions from models that have the same dependent variable and the same or similar independent variable candidates, as opposed to model ...
3
votes
2answers
38 views

Understanding Weighted learning in Ensemble Classifiers

I'm currently studying Boosting techniques in Machine Learning and I happened to understand that in Algorithms like Adaboost, each of the training samples is given a weight depending on whether it was ...
3
votes
0answers
50 views

Is linear regression on the trees of XGBoost (rather than taking their mean) useful/popular?

Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the ...
2
votes
2answers
586 views

Is there an R package for Locally Interpretable Model Agnostic Explanations?

One of the researchers, Marco Ribeiro, who developed this method of explaining how black box models make their decisions has developed a Python implementation of the algorithm available through Github,...
2
votes
2answers
965 views

Taking average of multiple neural networks?

I'm fitting a neural network using a very small data set, so try splitting the data into training and validation sets. (there is a separate test set) If I split training/validation randomly multiple ...
2
votes
2answers
263 views

Combining Different Models

I have N models, each of them are used to predict on a set of data. I am currently combining their predictions by averaging across rows. Need suggestions on combining their predictions. End goal is to ...
2
votes
2answers
379 views

How to measure the correlation of different algorithms

In stacked generalization, several algorithms are trained on the training set (i.e. at layer 1) and their predictions are then stacked using a layer 2 model. In many documentations, it is said that it ...
2
votes
1answer
171 views

Where does the random in Random Forests come from?

As the title says: Where does the random in Random Forests come from?
2
votes
2answers
99 views

Hyperparameter optimization, ensembling instead of selecting with CV criteria

While burning CPUs performing a CV selection on a thin grid put on some hyperparameter space. I am using the `scikit-learn' API, for which the end result is a single point on the hyperparameter space, ...
2
votes
1answer
240 views

Why Extra-trees should only be used within ensemble methods?

I was reading scikit-learn documentation for Extremely Randomized Trees and I found this warning: Warning: Extra-trees should only be used within ensemble methods. Why is that?
2
votes
1answer
2k views

EasyEnsemble explaination

Could someone please explain how the EasyEnsemble algorithm works? Im using it for a prediction model for imbalanced minority class. Please don't refer me to this paper, as it makes no sense to me. ...
2
votes
2answers
360 views

Is there any way to plot ROC curve for Ensemble hard voting classifier?

I am working on a multi-class text classification problem and performing an Ensemble learning for text classification. I chose hard voting as ensemble technique. I tried to plot ROC curve for my ...
2
votes
1answer
74 views

Stacked Model performance?

I am currently working with a dataset that seems very easily separable and I have an accuracy of 99% for SVM (NN-98%, RF-98%, DT-96-97% and I have checked for leakage & overfitting). As part of my ...
2
votes
1answer
23 views

is it possible to use the train result as another feature and retrain?

is it possible to use the train result as another feature and retrain? for example I make prediction with classification and add this result to the table and train xgboost?
2
votes
2answers
780 views

Random forest vs majority voting

I'm using spark with scala to implement majority voting of decision trees and random forest (both are configured in the same way - same depth, the same amount of base classifiers etc.). Dataset is ...
2
votes
1answer
120 views

How are Decision Trees averaged in Random Forest?

We all know that a Random Forest is an ensemble of Decision Trees, whose results are averaged. Every source I find simply talks about "averaging trees", but how does this "averaging of trees" happens?...
2
votes
1answer
30 views

How to approach model reporting task

I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc. The end goal is to predict churn rates ...
2
votes
1answer
689 views

Pick a model from multiple models using a decision tree

Let us say, I have 4 classification models on a training data set of various examples. Now, I want to choose which 1 out 4 models (or what combination of the 4 ...