People who code: we want your input. Take the Survey

# Questions tagged [ensemble-modeling]

In machine learning, ensemble methods combine multiple algorithms to make a prediction. Bagging, boosting, and stacking, are some examples.

42 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2k views

### How to tune weights in Voting Classifier (Sklearn)

I am trying to do the following: ...
38 views

### Understanding Weighted learning in Ensemble Classifiers

I'm currently studying Boosting techniques in Machine Learning and I happened to understand that in Algorithms like Adaboost, each of the training samples is given a weight depending on whether it was ...
50 views

### Is linear regression on the trees of XGBoost (rather than taking their mean) useful/popular?

Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the ...
38 views

### How to improve model performace when model shows a systemic pattern in residues

I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
46 views

### How to estimate the marginal distribution of a class with respect to one predictor in a classification task?

I have a dataset with a binary dependent variable $y \in \{0,1\}$ and a set of predictors $x1,x2,..,t$. Here, $t$ is the time in minutes (in 24 hrs, that is $t \in (0,1440)$). I want to estimate the ...
23 views

### Is there a universal convergence rate when stacking models/experts?

It's fairly common to see people stacking different models when chasing marginal gains in contexts such as Kaggle competitions or the Netflix challenge. I would like to know about the mathematics ...
1k views

### Training an ensemble of small neural networks efficiently in TensorFlow 2

I have a bunch of small neural networks (say, 5 to 50 feed-forward neural networks with only two hidden layers with 10-100 neurons each), which differ only in the weight initialization. I want to ...
78 views

### Averaging CNN perform worse than boosting

I'm trying to solve Quora Question Pairs with model stacking. My first layers are: CNN trained to predict the same target as whole model should "Magic features" like question frequency in ...
24 views

### Making an ensemble model for high F1 score

I presently have 2 algorithms that have a numerical output. Using a threshold of 0.9, I get the classification output. Let's say they are: P (high precision, low recall) R (high recall, low precision)...
8 views

### Large general-purpose model vs ensemble of many smaller models

I am reading this paper - https://arxiv.org/abs/1503.02531v1 - devoted to knowledge distillation in neural networks. One interesting approach is mentioned in this paper in sections ...
37 views

### How does stacking help Bias and Variance?

How does stacking help in terms of bias and variance? I have a hunch that stacking can help reduce bias but i am not sure, could someone refer to a paper?
76 views

### Stacking - Appropriate base and meta models

When implementing stacking for model building and prediction (For example using sklearn's StackingRegressor function) what is the appropriate choice of models for the base models and final meta model?...
18 views

### Selecting models for ensemble from large group of models with high uncertaintly

I'm in a situation where many models have been created, and I have their cross-validation performances as well as performance on test data. I need to select models for inclusion in a simple bagging ...
14 views

### General practices for building an incremental learning model which never forgets?

I'm new to datascience and appreciate your sage advice! I need to build an incremental learning model, and I know there's a lot that goes into something like that, but I'd like to highlight the most ...
20 views

### Decision Boundary with random new observations vs observations from test set

I'm trying to plot decision boundary for Decision Tree classifier. Classifier is trained on training set, and decision boundary (contour) using random new observations and observations from test set ...
64 views

### Can bagging ensemble consist of heterogeneous base models?

Bagging or bootstrap aggregation seems to make sense for time series forecasting using an ensemble because bagging randomizes subsets of the data with replacement. However, I've only seen bagging used ...
22 views

### Creating a sub-model from pre-trained model

I have a pre-trained model having the following architecture: ...
9 views

### Domain specific language to describe ensemble model

I'm looking for some tool/library/widely used approach to describe Hierarchical Model structure like ensemble: It's absolutely straightforward how to do it with simple ensemble like It can be ...
13 views

### How to Predict Employee count of businesses using Keras classifiers

I am trying to predict the amount of employees a business has based on a set of input variables. I am using things like the business's age, transaction details, geographic location, business structure ...
13 views

### Ensemble learning for multiple hypothesis classes

Just to confirm if the following description falls in the category of ensemble learning. Suppose given a training set $D=\{(X,Y)\}$ we are asked to train a regressor. But now the way we do it is to ...
19 views

### Would it make technical sense to model its two known subelements separately (CLV example)?

Let's assume Customer Lifetime Value (CLV) is defined as average basket x frequency. Option A is to build model predicting CLV ...
449 views

### checking model stability - Performance for different class

I tried to do multi-class classification problem. The goal is to predict whether the match will be won by HomeTeam, AwayTeam or Draw. I did feature engineering from the attributes and finally came up ...
198 views

### Ensemble Method using XGBoost and RotationForest python

How can I create an ensemble model using XGBoost and Rotation Forest in Python?
230 views

### Voting classifier using grid search for Time Series

I have three models: Arima Auto ARIMA Double Exponential Smoothing I would like to apply an ensemble method - a voting method and allow the classifier to learn weights for these three models. I ...
507 views

### Combining image and scalar inputs into a neural network

I'm looking at the best way of combining CNN with image input and a scalar value. I know that one of the ways is to concatenate flatten layer with this scalar value. But flatten layer consist for ...
82 views

### heterogeneous input features for estimators in sklearn.ensemble.VotingClassifier

I would like to do ensemble on my model. Two of them are SVM and XGBoosting. SVM could not tolerate null value and XGB can do it. So I have different features for each of them. but when ...
9k views

### How to avoid memory error with Pandas pd.read_csv method call with GridSearchCV usage for DecisionTreeRegressor model?

I have been implementing a DecisionTreeRegressor model in Anaconda environment with a data set sourced from a 20 million row, 12-dimensional CSV file. I could get the chunks off of the data set with ...
58 views

### AdaBoost - Ensemble model perform poor than best weak classifier

Can Adaboost's ensemble classifier perform worse than the best of the weak learners considered? If so when in what case of weak learner the ensemble learning does not perform better?
222 views

### Retrieve user features in real time from UserId for prediction

Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude/longitude, and time/date. Here is the proposed ...
484 views

### Compare probability estimates of two classifiers

Consider two binary classifiers A and B. Suppose that both A and B are predicting the same target, but that A is trained on a subset of the data for which a different set of features is available than ...
15 views

### How to make an lstm ensemble with different input shapes using keras

This is what I got so far for making an lstm ensemble with one model input for each of the lstm models and for the ensemble model and it works perfectly. ...
14 views

### Hybrid Ensembling

I read a paper recently where researchers were trying to do classification using ensembling methods. I first read about the concept of BOOSTED-STACKING there.For those who dont know , I can give a ...
14 views

### How to interpret gradient descent in boosting ensembles?

I struggle to grasp the role of gradient based optimization in boosting ensembles. As far as I understand boosting means combining a bunch of estimators (of the same types, usually decision trees) ...
43 views

### How do I proceed with model selection?

Only data scientist in an organization and I could really use a sounding board here. In Phase One of a project I deployed four models and served their average as the prediction. I used Random Forest ...
46 views

### KL Divergence between Predictions and Ground truth

I've got four (non-linear, tree-based) models in production and using the average of them as the served prediction. We get ground truth data immediately. During training the optimized candidate models ...
21 views

### Output all 1 when try to use ensemble CNN on MNIST dataset

...
10 views

According to this article (which references the original paper), this is the SAMME algorithm for multiclass classification using Adaboost: I would like to understand what is this term in step ...
27 views

### Model with 2 datasets: combine time series data and statistics

I am new to data science modelling so apologies if using wrong terminology in advance. I have a standard time series dataset of historical prices which is used to train/test a simple Random Forest ...
54 views

### What is the best way to use Early Stopping in an ensemble (stacking) model?

I have a training and a test dataset. I would like to use the output of Model A in an ensemble model. However, I would like to use early stopping. Usually, I would create Model A for each K-fold (on ...
661 views

### XGBoost Log Loss different from GridSearchCV Log Loss

I have a classification problem where i am trying to predict if the data returns a 1 or 0. So you're classic binary classification. I have my set of data that I have split into the dependent variables ...
I'm trying to create a k-Nearest-Neighbor based model of 76-dimensional input data $I$ and 44-dimensional output data $O$. Through domain knowledge I know that only certain input dimensions are ...