I am trying to build a majority vote system for 3 Neural Networks, and I came across the concept of Bagging method. Actually, I want to use neural networks as weak learners (I know it's debatable, but some papers have tried it and I want to try it too).

For more information about the voting system I tried to construct/constructed, please read the following thread (The softmax Layer is better the the function in the thread, because the majority function only gives equivalent accuracies of the 3 NNs, but doesn't improve the overall accuracy).

I read that bagging can improve the overall accuracy of the weak learners, but as you can see I only have 3 learners and there aren't any clear information about bagging with Neural Networks. I only found after some reading that I can use ensemble learning with Neural networks by using the output of the trained NNs in a linear fashion. If I want to detail this into steps, I would write:

  1. Divide Dataset into training and validation sets
  2. From the Training sat, construct 3 bootstraps samples ?, Or I need more?
  3. Train/Develop the 3 neural Networks on the 3 bootstraps samples?
  4. Test the Neural Networks on the validation/test set
  5. How can I join decisions of the NNs by bagging?

If possible, I need your insight on these implementation steps and to know if they follow the best practices of bagging with NNs.

PS: I just started reading about bagging and boosting, so I apologize for any conceptual mistakes and contradictions I might have said.

Best regards,


1 Answer 1


I think you have the right general idea.

  1. Divide the dataset into training and test.

  2. Divide the training dataset into k folds, call this M.

For each iteration of hyperparameter tuning (whatever you choose to tune);

For each fold i in M;

  1. Set the validation set to be equal to fold i, and the (inner) training set to be equal to all folds that are not i.

  2. Using the inner training set, generate N bootstrap resamples. Fit a single neural network model to each bootstrap resample, generating N models.

  3. Predict the validation set using each of the N neural network models.

  4. Count the votes of each neural network model. Set the final prediction for an observation in the validation set to whatever has the most votes as predicted by your N models. You should have a matrix of m observations by N predictions. Since you are actually classifying objects, try to keep N odd so that there are no ties. Alternatively, don't classify objects, and instead generate probabilities from your N models for each observation in the validation set. Then, calculate the mean predicted probability for each observation in your validation set, and choose a threshold for classification based off the costs of false positives/false negatives (probably superior, but I won't discuss this).

  5. Calculate your error using your final predictions from step 6 and the actual ground truth labels from the validation set. Save these error scores.


  1. Calculate the average error from all of the folds. If this average error is lower than a previous set of hyperparameters error, then save these hyperparameters from this iteration of tuning.


  1. Generate N bootstrap resamples of the entire training set (the training set created in step 1) and fit a single neural network model to each of the N bootstrap resamples, using the optimal set of hyperparameters that you found above for each neural network model.

  2. Generate predictions on the test set (created in step 1) using each of your N models. Set final predictions for the test set using the exact same methodology you used in step 6.

  3. Calculate the error using your final predictions from step 10 and the ground truth labels found in the test set. This is your final, unbiased estimate of model performance.

In general, the more bootstrap resamples you can generate, the better. However, since you are fitting neural networks you might need to be conservative with how many models you decide to bag due to computational costs. I recommend setting the number of bootstrap samples, N, to be as large as possible while being mindful of your own computational/time constraints.

EDIT: In response to the comments in which we require a faster method, one can replace step 2 (and delete step 3) and instead divide the training dataset into an inner training set and a validation one. We then validate our model to a single validation set rather than k sets, which has the potential to increase the likelihood of overfitting to a specific validation set (though since we are presumably dealing with a large dataset, it probably will be fine).

All other steps should remain the same, though there is clearly no loop over folds anymore. In step 8, there will also be no need to calculate the average error since we will only have one estimate of model error from the single validation set.

  • $\begingroup$ Waw, this needs more work than I've expected. And it's true, it might be time/resource consuming to implement. I think the part I didn't expect to see because it's the main reason why it might need resources to run, is working with k-folds and looping on each fold, is this necessary? or only to choose the best hyperparameters for the models ? $\endgroup$
    – U. User
    Aug 28, 2019 at 21:04
  • $\begingroup$ Instead of using k fold, you can simply split the training set into inner training set/validation for the hyperparameter selection (i.e. tune to a single validation set rather than k sets in cross validation). That will definitely lessen the time and computational resources needed but you will trade for a higher probability to overfit on a particular validation set. If you have lots of data though then k fold cv could probably be dropped for a single train/val instead. $\endgroup$
    – aranglol
    Aug 28, 2019 at 22:10
  • $\begingroup$ Considering that you are fitting neural networks, I can see that k fold cv might actually be a bit overkill since typically you will have many observations such that using a single split is still reliable. Otherwise if you aren't tuning your models/preprocessing your data you can likely do away with the validation set and skip straight to the test set. However, if you are going to tune then I don't see any other ways to save computational resources while still maintaining unbiased evaluation, besides potential parallel code. $\endgroup$
    – aranglol
    Aug 28, 2019 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.