67
votes
Accepted
GBM vs XGBOOST? Key differences?
Quote from the author of xgboost:
Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. ...
- 4,366
41
votes
Accepted
Adaboost vs Gradient Boosting
Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion.
Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more ...
- 6,452
20
votes
GBM vs XGBOOST? Key differences?
In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in ...
- 2,595
15
votes
GBM vs XGBOOST? Key differences?
One very important difference is xgboost has implemented DART, the dropout regularization for regression trees.
References
Rashmi, K. V., & Gilad-Bachrach, ...
- 1,390
8
votes
Accepted
Is there any difference between a weak learner and a weak classifier?
A weak learner can be either a classification or a regression algorithm:
Boosting (Schapire and Freund 2012) is a greedy algorithm for fitting adaptive basis-function models of the form in ...
- 5,490
7
votes
Accepted
Sklearn Aggregating Multiple Fitted Models Into A Single Model? (binary classification)
Despite the downvote, the question is clear, and a common one I'm sure most stumble across after doing machine learning work for some time.
The goal was to make a stronger predictive model from ...
- 239
7
votes
Is it possible to build ensemble models without a decision tree?
all the ensemble models I came through so far use/described using the decision tree.
Random Forest is the "ensemble version" of decision trees. It's a commonly used ensemble method because ...
- 25.9k
6
votes
Taking average of multiple neural networks?
I think even this method is also called Ensemble Method.
How could I conclude that?
You might have heard about this algorithm named Random Forest,
what does it do? It take data randomly at row level ...
- 2,392
6
votes
Accepted
What is the meaning of the term "pipeline" within data science?
A pipeline is almost like an algorithm, but at a higher level, in that it lists the steps of a process. People use it to describe the main stages of project. This could include everything from ...
- 15.1k
6
votes
Accepted
SHAP value can explain right?
I guess what you meant by correlation between SHAP values is "SHAP Interaction Value".
SHAP value is a measure how feature values are contributing a target variable in observation level. Likewise ...
- 874
5
votes
What does Negative Log Likelihood mean?
This answer correctly explains how the likelihood describes how likely it is to observe the ground truth labels t with the given data ...
- 602
5
votes
Is there an R package for Locally Interpretable Model Agnostic Explanations?
Yes, there is now a port to R, which is available here.
It purports to provide LIME explanations for any classifier that implements a predict() method accepting a <...
- 51
5
votes
What are the individual models within a machine learning ensemble called?
I am not aware of a specific definition. Wikipedia does not mention such a term either. I would prefer "components", "individual/constituent models", or something like that.
If you definitely want to ...
- 151
5
votes
Accepted
Are "Gradient Boosting Machines (GBM)" and GBDT exactly the same thing?
Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the ...
- 6,370
5
votes
Accepted
Probability that ensemble model is correct based on accuracies of its classifiers
It's not the actual data, it's the probabilities. So you should consider all the scenarios of voting.
For the Ensemble to be correct,
Either any two or all the three should be correct
=$[m_1*m_2*(1- ...
- 5,759
5
votes
Accepted
What ML model for regression given tabular AND image data?
Is there a way to combine a random tree/XGB and a CNN for regression ?
An MLP that is integrated into the main architecture would actively learn the best patterns to extract based on the regression ...
- 836
4
votes
Accepted
Is there an R package for Locally Interpretable Model Agnostic Explanations?
I think you're talking about the lime Python package. No, there is no R port for the package. The implementation for the localized model requires enhancements to ...
- 3,610
4
votes
Accepted
Improving classifier performances in R for imbalanced dataset
You should try compensating for the imbalanced data and then can you try a lot of different classifiers. Either balance it out, use SMOTE to interpolate (this always struck me as too magical), or ...
- 1,663
4
votes
Accepted
How predictions of level 1 models become training set of a new model in stacked generalization.
You have created a model pipeline and must run all trained models ("lower level" ones first) in order to make a prediction on new data using the stack.
With test data set, it is slightly easier, ...
- 29.2k
4
votes
Combining Different Models
Part of the problem lies in how much data you have. To create a second level of complexity, you ideally want to use a holdout data set to decide the right combination for the model predictions. If you ...
- 311
4
votes
Accepted
Questions on ensemble technique in machine learning
Instead, model 2 may have a better overall performance on all the data
points, but it has worse performance on the very set of points where
model 1 is better. The idea is to combine these two ...
- 308
4
votes
Accepted
Why Extra-trees should only be used within ensemble methods?
In a random forest tree, a random subset of features is available for consideration at each split. Extra-trees takes this a step further by using a random threshold at each split. The idea is that a ...
4
votes
Accepted
Is there any implementation of Extended Isolation Forest algorithm in R/Python?
There is a package on Github called "Extended Isolation Forest for Anomaly Detection", I used it a couple months ago and it seemed to work. For how accurate or how buggy it is, I'm not sure but if ...
- 3,460
4
votes
How are Decision Trees averaged in Random Forest?
It depends on the type of variable that the Random Forest is predicting, and perhaps on the specific implementation of Random Forest. Following is an overview of the simplest techniques.
Continuous ...
- 2,772
4
votes
Accepted
How to optimize hyperparameters in stacked model?
I believe the most common way involves some slight data leakage during the training step that is often ignored. The "correct" way involves giving up more training data but many have empirically ...
- 2,206
4
votes
Accepted
If you are making a ensemble model does training data on base models have to be different from one another
When ensembling, you need some method of introducing diversity into your models (otherwise all your models will make the same prediction, so ensembling them won't improve the results). Using different ...
- 1,342
3
votes
Accepted
Why some people add results from PCA and other dimensional reductions techniques as features
This is feature engineering. You just give the algorithm another look at the data, from another point of view. It often helps to understand better data when you have different point of views.
For ...
- 136
3
votes
Accepted
How to measure the correlation of different algorithms
For regression tasks correlation will be simply the correlation between the predicted values, for binary classification it will be correlation between predicted probabilities. In multiclass ...
- 378
3
votes
Ensemble Probabilities of the different models
I think it can be done by using this command at the time of prediction, giving example in R
...
- 2,392
3
votes
What are the individual models within a machine learning ensemble called?
I have heard people calling them "weak learners" many times, but this is only when they are not very acurate themselves.
- 6,106
Only top scored, non community-wiki answers of a minimum length are eligible