# Tag Info

6

all the ensemble models I came through so far use/described using the decision tree. Random Forest is the "ensemble version" of decision trees. It's a commonly used ensemble method because it's built in the algorithm itself. However ensemble methods are much more general than decision trees and can be used with any learning method, for example by ...

4

I think the most highly-referenced source for these terms is Horizontal and Vertical Ensemble with Deep Representation for Classification (Xe, Xu, Chuang 2013). That would be the best place to get a technical answer to your first question. For purposes of searching, you could also look into "Stacking" as a synonym for Vertical ensembling. I will provide some ...

3

It is absolutely a way to improve your classifier's accuracy. Actually a "strong" enough classifier such as a neural network could be able to learn by itself these clusters. However, you would need a substancially deeper network. The "smartest" way to do this, if you know there are many groups/clusters in your data is to actually perform a 2-steps process: ...

3

Let's say we have a set of 40 numbers from 1 to 40. We have to pick 4 subsets of 10 numbers. Case 1 - Bagging - We will pick the first number, put it back, and then pick the next. This makes all the draw independent and consequently have very little correlation. So, if you make a Tree on the first 10 samples and another Tree on the next, both the trees will ...

2

I think that the reason for this to happen is that tree-based methods have problems with linear problems. This is because tree-based methods do partitions of the variables, and not on combinations of the variables. To fit a linear regression, a tree-based method will have to do a lot of partitions to obtain low error. However, in principle, using enough ...

2

We can answer this overarching question by exploring a couple sub-questions: What are the properties of popular averaging formulae? Geometric: $\space$ $\space$ $\begin{equation} \bigg(\displaystyle \prod^N_{i=1} x_{i}\bigg)^{\frac{1}{N}} \end{equation}$ We can see that any set that contains a zero has a geometric mean of zero. Also, processing ...

2

At the highest level of abstraction, the answer is yes. You can send a set of values to be scored to every model in an ensemble, and then combine the resulting of set of scores into a single score according to a predetermined formula. Formally speaking, every transaction follows the same path through the system, so no decision is involved.

2

I think you confused some technical names. Cross-Validation is the name of the procedure, and it has some techniques or approaches such as k-fold cross-validation, train test split, etc. All are techniques to measure the performance of a model. In your case, you have the first model that is assessed using 10-fold cross-validation and has an f1-score of 0.941,...

1

You can indeed use other weak learners (as the components of an ensemble are commonly called) than just decision trees. That said, decision tree ensembles are most widely used, especially gradient boosted trees and random forest. Sometimes, other ensembles are just a conceptual tool to facilitate analysis of algorithms, like when you're trying to understand ...

1

The issue of using any linear model (a polynomial regression is a particular case of a linear model, with polynomial features), is that an ensemble of linear models is still a linear model. So, the family of models to optimize from given by the boosted polynomial regression and the single polynomial regression are the same. This doesn't happen with trees, as ...

1

Yes, I think that's a sound approach and a good way to compare different systems. A ROC curve comparison is usually more informative than the raw performance scores, but it's still quite general. In case you want to observe even more detail, you could also try to look at specific groups of instances. One way to do that is to count for every instance how ...

1

It does not support at the time (it will come just as xgboost did not have to have it) Given thats its a boosting method in the first place one can ask whats the history of xbgoost and subsequent cat and lgboost. XGBoost implementation of gradientboosting did not handle categorical features because it did not have to, it was sufficient enough as it was. ...

1

You can not feed your network with two inputs with different number of samples, and this also does not make sense. You have 2 inputs with shape (502,) and (1002,) (You have said you want to extract features also from your second dataset). Let's consider the batch size is 1 for the sake of simplicity. So the model takes one sample each time to move it through ...

1

You can test each base classifier on a hold-out dataset and come up with a performance metric say accuracy for each model. You can then use each model's accuracy as a weight when combining predictions in the ensemble. Here is a useful article.

1

It really depends on how other models behave on your particular dataset, but generally, you should check 2 things: Predictions from ensembled models are not highly correlated and other models also perform well on your dataset. It's better to use a model from another family. So if you are using XGboost, its better to add some model like SVM than a catboost (...

1

My intuition would be to try to integrate the information about the products directly in the original model. Typically the possible products in a shipment can be represented as boolean features (one hot encoding), but this part might need some feature engineering if there are too many different products: simple option: only a small set of features ...

1

You can make the same question with every Machine Learning algorithm, and still the answer will remain very similar. What's the advantage of Linear regression over Decision Trees? To answer this you could define them mathematically. In your case, the mathematical definition seems easy: weighted mean or geometric mean. When do any model works better by any ...

1

That is an empirical question. The answer will change for different models and different datasets. The best approach would use cross validation to see which ensembling technique has the best score on the evaluation metric for the given data.

1

AdaBoost or Adaptive Boost is a boosting ensemble model which works by learning from it's previous mistakes, ie: misclassified data points. We specify the number of decision trees to be generated while training and during each training step, it calculates the following : The weighted error rate of the trained decision tree The decision tree's weight in ...

1

Is an interesting question. You are solving a optimization problem $max$ $AUC(\alpha_1*X_1 + \alpha_2*X_2,Y)$ $s.t.$ $\sum^2_{i=1}\alpha = 1$ You are maximizing the AUC using the models 1 and 2 (which are $X_1$ and $X_2$) subject to the sum of weights is 1. You may any optimization algorithm, even Excel's solver to do this.

1

You want to ensemble your two algorithms. The way to do that is to not just use e.g. sklearn's precision and recall metric functions, but to actually obtain probabilities for being positive (most models can output this with a simple function call), and take the average of those probabilities between your two (or more) binary classifiers. Then, test a range ...

1

Bagging main goal is to minimize variance of your model. Basically, if you have a model that is on average pretty accurate but inconsistent (meaning, it does well for a given data set, poorly generalizations) then bagging may be a way to produce a more consistent estimators. Decision trees are the common example of this because they are the canonical high ...

1

You can try SHAP which visually explains the output of (many) machine learning model(s) including LightGBM and XGBoost. However, please note that it will not give you the entire Ensemble Model (Trees) as picture. Further note that it doesn't work for RandomForest

1

(adding to what's said above by @David), The short answer is that, You can't expect the tree based models to Extrapolate... Had asked on Slack (quoting miguel_perez)and this was the reply, realize that in your example you are aproximating a line with an staircase. Even discarding other errors suspect number one would be not enough data points. Trees are ...

1

Based on your description, it looks like different models have different biases. Two important questions: do you have any data imbalance problem? What kind of models you are using? Using stacking based classifier is beneficial if you have different biases. Try to use a simple stack based classifier. For your level-1 classifier, use different models (e.g. SVM-...

1

Feature-Weighted Linear Stacking might be what you are looking for. FWLS combines model predictions linearly using coefficients that are themselves linear functions of meta-features. In your example you can use the meta-feature "Does A label the example as True?"

Only top voted, non community-wiki answers of a minimum length are eligible