67
votes
Accepted
GBM vs XGBOOST? Key differences?
Quote from the author of xgboost:
Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. ...
- 4,296
40
votes
Accepted
Adaboost vs Gradient Boosting
Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion.
Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more ...
oW_♦
- 6,264
20
votes
GBM vs XGBOOST? Key differences?
In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in ...
- 2,517
16
votes
Accepted
what is the difference between "fully developed decision trees" and "shallow decision trees"?
[Later edit - Rephrase everything]
Types of trees
A shallow tree is a small tree (most of the cases it has a small depth). A full grown tree is a big tree (most of the cases it has a large depth).
...
- 4,643
15
votes
GBM vs XGBOOST? Key differences?
One very important difference is xgboost has implemented DART, the dropout regularization for regression trees.
References
Rashmi, K. V., & Gilad-Bachrach, ...
- 1,350
8
votes
Accepted
Is there any difference between a weak learner and a weak classifier?
A weak learner can be either a classification or a regression algorithm:
Boosting (Schapire and Freund 2012) is a greedy algorithm for fitting adaptive basis-function models of the form in ...
- 5,360
7
votes
Accepted
What does Negative Log Likelihood mean?
Likelihood function is the product of probability distribution function, assuming each observation is independent. However, we usually work on a logarithmic scale, because the PDF terms are now ...
- 3,520
7
votes
Accepted
Sklearn Aggregating Multiple Fitted Models Into A Single Model? (binary classification)
Despite the downvote, the question is clear, and a common one I'm sure most stumble across after doing machine learning work for some time.
The goal was to make a stronger predictive model from ...
- 239
7
votes
Is it possible to build ensemble models without a decision tree?
all the ensemble models I came through so far use/described using the decision tree.
Random Forest is the "ensemble version" of decision trees. It's a commonly used ensemble method because ...
- 25k
6
votes
Ensembling vs clustering in machine learning
Short answer: Ensembling and clustering are completely unrelated techniques.
Ensembling: Combine the strengths of many diverse models. Ensembles generally do not involve training models on separate ...
- 4,139
6
votes
2 stage ensemble -- CV MSE valid in 1st stage but not in 2nd
To clarify: you build one random forest on training data and get some results which seems to have no overfit, since CV and test results are similar. The second RF is built on the predictions of the ...
- 4,643
6
votes
Accepted
Is SuperLearning actually different to stacking,or are they essentially the same thing?
So Ensemble Learning is essentially using multiple learning algorithms and providing the best predictive performance considering all of them. This gives a better detailed description. Now, Ensemble ...
- 2,316
6
votes
Taking average of multiple neural networks?
I think even this method is also called Ensemble Method.
How could I conclude that?
You might have heard about this algorithm named Random Forest,
what does it do? It take data randomly at row level ...
- 2,382
6
votes
Accepted
What is the meaning of the term "pipeline" within data science?
A pipeline is almost like an algorithm, but at a higher level, in that it lists the steps of a process. People use it to describe the main stages of project. This could include everything from ...
- 14.7k
6
votes
Accepted
SHAP value can explain right?
I guess what you meant by correlation between SHAP values is "SHAP Interaction Value".
SHAP value is a measure how feature values are contributing a target variable in observation level. Likewise ...
- 866
5
votes
Accepted
Where does the random in Random Forests come from?
For each tree you randomly select from the variables that you can use to split tree nodes. Generally you randomly select 1/3 of the variables per tree.
- 4,139
5
votes
What does Negative Log Likelihood mean?
This answer correctly explains how the likelihood describes how likely it is to observe the ground truth labels t with the given data ...
- 582
5
votes
Is there an R package for Locally Interpretable Model Agnostic Explanations?
Yes, there is now a port to R, which is available here.
It purports to provide LIME explanations for any classifier that implements a predict() method accepting a <...
- 51
5
votes
What are the individual models within a machine learning ensemble called?
I am not aware of a specific definition. Wikipedia does not mention such a term either. I would prefer "components", "individual/constituent models", or something like that.
If you definitely want to ...
- 151
5
votes
Accepted
Are "Gradient Boosting Machines (GBM)" and GBDT exactly the same thing?
Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the ...
- 6,092
5
votes
Accepted
Probability that ensemble model is correct based on accuracies of its classifiers
It's not the actual data, it's the probabilities. So you should consider all the scenarios of voting.
For the Ensemble to be correct,
Either any two or all the three should be correct
=$[m_1*m_2*(1- ...
- 5,554
4
votes
Accepted
Is there an R package for Locally Interpretable Model Agnostic Explanations?
I think you're talking about the lime Python package. No, there is no R port for the package. The implementation for the localized model requires enhancements to ...
- 3,520
4
votes
Accepted
Improving classifier performances in R for imbalanced dataset
You should try compensating for the imbalanced data and then can you try a lot of different classifiers. Either balance it out, use SMOTE to interpolate (this always struck me as too magical), or ...
- 1,663
4
votes
Accepted
How predictions of level 1 models become training set of a new model in stacked generalization.
You have created a model pipeline and must run all trained models ("lower level" ones first) in order to make a prediction on new data using the stack.
With test data set, it is slightly easier, ...
- 28.6k
4
votes
Combining Different Models
Part of the problem lies in how much data you have. To create a second level of complexity, you ideally want to use a holdout data set to decide the right combination for the model predictions. If you ...
- 301
4
votes
Accepted
Why Extra-trees should only be used within ensemble methods?
In a random forest tree, a random subset of features is available for consideration at each split. Extra-trees takes this a step further by using a random threshold at each split. The idea is that a ...
4
votes
Accepted
How to optimize hyperparameters in stacked model?
I believe the most common way involves some slight data leakage during the training step that is often ignored. The "correct" way involves giving up more training data but many have empirically ...
- 1,954
4
votes
Accepted
If you are making a ensemble model does training data on base models have to be different from one another
When ensembling, you need some method of introducing diversity into your models (otherwise all your models will make the same prediction, so ensembling them won't improve the results). Using different ...
- 1,221
3
votes
Accepted
EasyEnsemble explaination
The toolbox only manage the sampling so this is slightly different from the algorithm from the paper.
What it does is the following: it creates several subset of data which are balanced. These ...
- 286
3
votes
Accepted
Stacked features not helping
As far as I understood stacking does not add features to the original data set. The point is to train several models on the training data and use their predictions on training data as input features ...
- 4,643
Only top scored, non community-wiki answers of a minimum length are eligible