Ok, so I have the following set up:
I have a binary classification problem and I am classifying events into signal and background. Ultimately, I want to count how many background events and how many signal events I have in an unknown data set.
I am using gradient boosted decision trees and I am training my model about 100 times, each time with random selected training and test set (which is called bootstrapping if I am correct).
Now I want to predict the class (signal or background) of instances in a new data set and I dont know the class. I use my ensemble of 100 trained models, so I get a prediction after each of the 100 rounds of training and testing. In the end I get 100 prediction scores (a number between 1 and 0) for each instance in the data set. Afterwards, I can average over the 100 outcomes and get a mean and standard deviation for every data instance.
After setting a cut value on the prediction score (everything below that value is background and everything above is signal), I can count how many events lie above and how many below and that gives me a number of signal and a number of background events.
Now (finally) my question: can I somehow include the error I gain due to averaging over the 100 outcomes into the error on the counting? Or is there a (Bayesian) way of not reducing the distributions to the mean and standard deviation but somehow continue with the prediction score distributions to obtain an value for number of background and signal counts?