I have 10 groups of biological experiments, all of size 100. I want to estimate experimental performance (success rate) of each groups of experiments, but have only ran experiments in two groups. My plan was to train a model on the 200 data points to predict experimental performance and use that model to predict experimental performance across the other 8 groups. Experimental performance here is defined as the fraction of successful experiments.

I've done 10 fold cross validation on a random forest, and the performance on the 2 groups is pretty good f1~.94 precision~.95, recall~.93. To estimate experimental performance, I trained 100 RF models on bootstrapped versions of the training data (This gives me some error bounds). But when I try to use a model to estimate experimental performance I under estimate the experimental performance on one of the groups that we have actually run experiments for. Looking into this further, I observe that no matter what i do, there are 8 datapoints that are always classified as false negatives, which is reducing the performance.

How should I handle these 8 points? Is it ok to just say, these could be issues with the data and move on? Or is there something I can do to try to correctly classify these points?

  • $\begingroup$ Performance here is defined as the number of predicted 1 values. If performance is greatest when you predict the most 1s, what incentive do you have to predict anything else? It seems like you will achieve the best performance by always predicting the outcome to be 1. $\endgroup$
    – Dave
    Commented Nov 26, 2023 at 16:09
  • $\begingroup$ Each group represents a set of lag experiments. A label of 1 means that experiment was successful, a label of 0 means failure. So we want as many successful experiments as possible. We have only run those two groups so far. We want to estimate the performance of the remaining experiments using this model. $\endgroup$ Commented Nov 26, 2023 at 16:21
  • $\begingroup$ I'll clarify the situation. Thanks $\endgroup$ Commented Nov 26, 2023 at 16:24
  • $\begingroup$ So then you make predictions about data for which you do not know the outcome. I’m not seeing an issue. $\endgroup$
    – Dave
    Commented Nov 26, 2023 at 16:25
  • $\begingroup$ I only know the outcome for the 2 groups, and even with high model performance, the experimental performance of one is the groups is underestimated. If the model can't correctly predict the groups I have data for how do I trust it's ability to predict for be groups? $\endgroup$ Commented Nov 26, 2023 at 16:47


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.