59 votes
Accepted

GBM vs XGBOOST? Key differences?

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. ...
user avatar
  • 4,016
37 votes
Accepted

Adaboost vs Gradient Boosting

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more ...
user avatar
  • 5,987
18 votes

GBM vs XGBOOST? Key differences?

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in ...
user avatar
14 votes

GBM vs XGBOOST? Key differences?

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, ...
user avatar
  • 1,320
14 votes
Accepted

what is the difference between "fully developed decision trees" and "shallow decision trees"?

[Later edit - Rephrase everything] Types of trees A shallow tree is a small tree (most of the cases it has a small depth). A full grown tree is a big tree (most of the cases it has a large depth). ...
user avatar
  • 4,418
8 votes
Accepted

Is there any difference between a weak learner and a weak classifier?

A weak learner can be either a classification or a regression algorithm: Boosting (Schapire and Freund 2012) is a greedy algorithm for fitting adaptive basis-function models of the form in ...
user avatar
  • 4,837
7 votes
Accepted

Sklearn Aggregating Multiple Fitted Models Into A Single Model? (binary classification)

Despite the downvote, the question is clear, and a common one I'm sure most stumble across after doing machine learning work for some time. The goal was to make a stronger predictive model from ...
user avatar
  • 229
7 votes

Is it possible to build ensemble models without a decision tree?

all the ensemble models I came through so far use/described using the decision tree. Random Forest is the "ensemble version" of decision trees. It's a commonly used ensemble method because ...
user avatar
  • 21.8k
6 votes

Ensembling vs clustering in machine learning

Short answer: Ensembling and clustering are completely unrelated techniques. Ensembling: Combine the strengths of many diverse models. Ensembles generally do not involve training models on separate ...
user avatar
  • 4,039
6 votes

2 stage ensemble -- CV MSE valid in 1st stage but not in 2nd

To clarify: you build one random forest on training data and get some results which seems to have no overfit, since CV and test results are similar. The second RF is built on the predictions of the ...
user avatar
  • 4,418
6 votes
Accepted

Is SuperLearning actually different to stacking,or are they essentially the same thing?

So Ensemble Learning is essentially using multiple learning algorithms and providing the best predictive performance considering all of them. This gives a better detailed description. Now, Ensemble ...
user avatar
  • 2,296
6 votes
Accepted

What does Negative Log Likelihood mean?

Likelihood function is the product of probability distribution function, assuming each observation is independent. However, we usually work on a logarithmic scale, because the PDF terms are now ...
user avatar
  • 3,380
6 votes

Taking average of multiple neural networks?

I think even this method is also called Ensemble Method. How could I conclude that? You might have heard about this algorithm named Random Forest, what does it do? It take data randomly at row level ...
user avatar
  • 2,352
5 votes
Accepted

Where does the random in Random Forests come from?

For each tree you randomly select from the variables that you can use to split tree nodes. Generally you randomly select 1/3 of the variables per tree.
user avatar
  • 4,039
5 votes

Is there an R package for Locally Interpretable Model Agnostic Explanations?

Yes, there is now a port to R, which is available here. It purports to provide LIME explanations for any classifier that implements a predict() method accepting a <...
user avatar
  • 51
5 votes

What are the individual models within a machine learning ensemble called?

I am not aware of a specific definition. Wikipedia does not mention such a term either. I would prefer "components", "individual/constituent models", or something like that. If you definitely want to ...
user avatar
  • 151
5 votes
Accepted

What is the meaning of the term "pipeline" within data science?

A pipeline is almost like an algorithm, but at a higher level, in that it lists the steps of a process. People use it to describe the main stages of project. This could include everything from ...
user avatar
  • 13.9k
5 votes
Accepted

SHAP value can explain right?

I guess what you meant by correlation between SHAP values is "SHAP Interaction Value". SHAP value is a measure how feature values are contributing a target variable in observation level. Likewise ...
user avatar
5 votes
Accepted

Are "Gradient Boosting Machines (GBM)" and GBDT exactly the same thing?

Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the ...
user avatar
5 votes
Accepted

Probability that ensemble model is correct based on accuracies of its classifiers

It's not the actual data, it's the probabilities. So you should consider all the scenarios of voting. For the Ensemble to be correct, Either any two or all the three should be correct =$[m_1*m_2*(1- ...
user avatar
  • 5,164
4 votes

What does Negative Log Likelihood mean?

This answer correctly explains how the likelihood describes how likely it is to observe the ground truth labels t with the given data ...
user avatar
  • 532
4 votes
Accepted

Is there an R package for Locally Interpretable Model Agnostic Explanations?

I think you're talking about the lime Python package. No, there is no R port for the package. The implementation for the localized model requires enhancements to ...
user avatar
  • 3,380
4 votes
Accepted

Improving classifier performances in R for imbalanced dataset

You should try compensating for the imbalanced data and then can you try a lot of different classifiers. Either balance it out, use SMOTE to interpolate (this always struck me as too magical), or ...
user avatar
  • 1,633
4 votes
Accepted

How predictions of level 1 models become training set of a new model in stacked generalization.

You have created a model pipeline and must run all trained models ("lower level" ones first) in order to make a prediction on new data using the stack. With test data set, it is slightly easier, ...
user avatar
  • 27.3k
4 votes

Combining Different Models

Part of the problem lies in how much data you have. To create a second level of complexity, you ideally want to use a holdout data set to decide the right combination for the model predictions. If you ...
user avatar
  • 301
4 votes
Accepted

Why Extra-trees should only be used within ensemble methods?

In a random forest tree, a random subset of features is available for consideration at each split. Extra-trees takes this a step further by using a random threshold at each split. The idea is that a ...
user avatar
3 votes
Accepted

EasyEnsemble explaination

The toolbox only manage the sampling so this is slightly different from the algorithm from the paper. What it does is the following: it creates several subset of data which are balanced. These ...
user avatar
  • 276
3 votes
Accepted

Stacked features not helping

As far as I understood stacking does not add features to the original data set. The point is to train several models on the training data and use their predictions on training data as input features ...
user avatar
  • 4,418
3 votes
Accepted

Why some people add results from PCA and other dimensional reductions techniques as features

This is feature engineering. You just give the algorithm another look at the data, from another point of view. It often helps to understand better data when you have different point of views. For ...
user avatar
  • 126
3 votes
Accepted

How to measure the correlation of different algorithms

For regression tasks correlation will be simply the correlation between the predicted values, for binary classification it will be correlation between predicted probabilities. In multiclass ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible