53

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. Specifically, xgboost used a more regularized model formalization to control over-fitting, which gives it better performance. We have updated a comprehensive tutorial on introduction to the model, which ...


34

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more weight on misclassified samples and less weight on correctly classified samples. The final prediction is a weighted average of all the weak learners, where more ...


14

[Later edit - Rephrase everything] Types of trees A shallow tree is a small tree (most of the cases it has a small depth). A full grown tree is a big tree (most of the cases it has a large depth). Suppose you have a training set of data which looks like a non-linear structure. Bias variance decomposition as a way to see the learning error ...


14

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in speed and memory utilization: Use of sparse matrices with sparsity aware algorithms Improved data structures for better processor cache utilization which makes ...


12

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, R. (2015). Dart: Dropouts meet multiple additive regression trees. arXiv preprint arXiv:1505.01866.


8

A weak learner can be either a classification or a regression algorithm: Boosting (Schapire and Freund 2012) is a greedy algorithm for fitting adaptive basis-function models of the form in Equation 16.3, where the $\phi_m$ are generated by an algorithm called a weak learner or a base learner. The algorithm works by applying the weak learner sequentially ...


7

Despite the downvote, the question is clear, and a common one I'm sure most stumble across after doing machine learning work for some time. The goal was to make a stronger predictive model from multiple trained models. Quote from my question: is it possible to aggregate these 30 fitted models into a single model? Answer: yes but there's no good ...


6

Short answer: Ensembling and clustering are completely unrelated techniques. Ensembling: Combine the strengths of many diverse models. Ensembles generally do not involve training models on separate sets of data -- it's the models themselves that are different. Generally the more diverse the models the better. For example, an ensemble might comprise the ...


6

To clarify: you build one random forest on training data and get some results which seems to have no overfit, since CV and test results are similar. The second RF is built on the predictions of the first RF and some other additional features. The error on CV training data is very low and on test data are very high. Now we will analyze the performance of the ...


6

So Ensemble Learning is essentially using multiple learning algorithms and providing the best predictive performance considering all of them. This gives a better detailed description. Now, Ensemble Learning can be broadly divided into multiple types like Boosting Bagging Stacking / Super Learning Stacking contains a bunch of algorithms along with a ...


6

I think even this method is also called Ensemble Method. How could I conclude that? You might have heard about this algorithm named Random Forest, what does it do? It take data randomly at row level and column level builds different trees and takes an average of it. It is also considered as one of the best algorithm for Prediction and Classification. Can ...


5

For each tree you randomly select from the variables that you can use to split tree nodes. Generally you randomly select 1/3 of the variables per tree.


5

Yes, there is now a port to R, which is available here. It purports to provide LIME explanations for any classifier that implements a predict() method accepting a type = 'prob' argument. I have not yet tested it myself.


5

I am not aware of a specific definition. Wikipedia does not mention such a term either. I would prefer "components", "individual/constituent models", or something like that. If you definitely want to find a "correct" term, one way to discover it (if it exists) is to look into an ensemble learning early paper. To arrive at one, a good way is to search for a ...


5

A pipeline is almost like an algorithm, but at a higher level, in that it lists the steps of a process. People use it to describe the main stages of project. This could include everything from gathering data and pre-processing it, right through to post-analysis of predictions. The pipeline is essentially a large chain of modules, which can be individually ...


5

I guess what you meant by correlation between SHAP values is "SHAP Interaction Value". SHAP value is a measure how feature values are contributing a target variable in observation level. Likewise SHAP interaction value considers target values while correlation between features (Pearson, Spearman etc) does not involve target values therefore they might have ...


5

It's not the actual data, it's the probabilities. So you should consider all the scenarios of voting. For the Ensemble to be correct, Either any two or all the three should be correct =$[m_1*m_2*(1- m_3) + m_1*(1-m_2)*m_3) + (1-m_1)*m_2*m_3] + [m_1*m_2*m_3]$ = [0.6*0.55*(1-0.45) + 0.6*(1-0.55)*0.45 + (1-0.6)*0.55*0.45] + 0.6*0.55*0.45 = 0.5505


4

Likelihood function is the product of probability distribution function, assuming each observation is independent. However, we usually work on a logarithmic scale, because the PDF terms are now additive. If you don't understand what I've said, just remember the higher the value it is, the more likely your model fits the model. Google for maximum likelihood ...


4

I think you're talking about the lime Python package. No, there is no R port for the package. The implementation for the localized model requires enhancements to the existing machine-learning code (explained in the paper), a new implementation for R would be very time consuming. You may want to take a look at this for interfacing Python in R. My suggestion ...


4

You should try compensating for the imbalanced data and then can you try a lot of different classifiers. Either balance it out, use SMOTE to interpolate (this always struck me as too magical), or assign weights. Here's a nice article walking through it with caret, which is what it appears you're using: http://dpmartin42.github.io/blogposts/r/imbalanced-...


4

You have created a model pipeline and must run all trained models ("lower level" ones first) in order to make a prediction on new data using the stack. With test data set, it is slightly easier, since you can store the predictions from the "level 1" models when testing them, and only run the final model across this stored data. In addition to your brief ...


4

Part of the problem lies in how much data you have. To create a second level of complexity, you ideally want to use a holdout data set to decide the right combination for the model predictions. If you use the training data from the models themselves to also combine the model output, you risk over fitting your final model. If you have a small data set, trying ...


4

In a random forest tree, a random subset of features is available for consideration at each split. Extra-trees takes this a step further by using a random threshold at each split. The idea is that a forest (an ensemble of trees) with a large number of trees that have learned something different from each other (due to the random nature of the features and ...


4

Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the optimization technique called gradient descent (Boosting Algorithms as Gradient Descent. Given this, you can boost any kind of model that you want (as far as I know)....


3

The toolbox only manage the sampling so this is slightly different from the algorithm from the paper. What it does is the following: it creates several subset of data which are balanced. These subsets are created by randomly under-sampling the majority class. That is what you are getting from the toolbox. To obtain what in the paper, you need to train an ...


3

As far as I understood stacking does not add features to the original data set. The point is to train several models on the training data and use their predictions on training data as input features to another model. First such kind of construction used logistic regression as a final ensemble and and class probabilities from each base learner as input ...


3

This is feature engineering. You just give the algorithm another look at the data, from another point of view. It often helps to understand better data when you have different point of views. For example, let's assume you want to learn the US road directions to a simple ML model. You give him all examples from roads 1 to 100 mapping to 0 (if the road is ...


3

For regression tasks correlation will be simply the correlation between the predicted values, for binary classification it will be correlation between predicted probabilities. In multiclass classification you can find correlation between predicted factor variables using the hetcor package in R


3

I think it can be done by using this command at the time of prediction, giving example in R #To predict with probabilities testSet$pred_rf_prob<-`predict(object,model_rf,testSet[,predictors],type='prob')` To take average of the predictions: testSet$pred_avg<-(testSet$pred_rf_prob$Y+testSet$pred_knn_prob$Y+testSet$pred_lr_prob$Y)/3 This Link , might ...


3

Instead, model 2 may have a better overall performance on all the data points, but it has worse performance on the very set of points where model 1 is better. The idea is to combine these two models where they perform the best. This is why creating out-of-sample predictions have a higher chance of capturing distinct regions where each model ...


Only top voted, non community-wiki answers of a minimum length are eligible