I have a large dataset approximately 150k rows and 1500 of positive labels on which I can train my model for binary classification.

And also I have the other dataset which is smaller and is comprised from 80k rows and 100 positive labels.

The problem is that I can't train model on the small dataset because it results in bad quality. And the model trained on the large dataset can provide more stable outcomes for the second case due to the targets and domain similarity. Unfortunately, this model probability calibration is terrible for the small dataset.

So the question is, is it valid to apply the following pipeline: to train logistic regression on the large dataset -> to recalibrate it on the small dataset with isotonic regression, for example -> to score test data from the same source as the small dataset

I've implemented this pipeline and it looks good but I doubt whether it is correct. I've seen this post but I'm not sure it's about the same problem

  • $\begingroup$ Seems reasonable, i dont see a reason why it would be incorrect $\endgroup$ Commented Apr 28, 2022 at 11:19
  • $\begingroup$ thanks, appreciate $\endgroup$ Commented Apr 28, 2022 at 11:58
  • $\begingroup$ What is the relation between the large and small datasets? Are they coming from the same source and do they have the same variables? $\endgroup$ Commented Apr 28, 2022 at 15:19
  • 1
    $\begingroup$ @RalphWinters this is a good question. We have a lot of datasets from previous projects and they are more or less from the same domain. And so to prepare model for the small dataset we decided to find the pretrained model. The pretrained model corresponding to large dataset provides by accident the best result on the small dataset but calibration is just a horizontal line. We collect data from the same source the variables are the same but the targets can come from different samples of one domain. For example different users groups $\endgroup$ Commented Apr 28, 2022 at 16:56
  • 1
    $\begingroup$ This is a tricky problem. Basically you are assuming the previous datasets are from the same populations. That may account for some of the 'oddness'. The only other thing I would suggest it to start by exploring the frequencies and the means of all of the groups (without doing the modeling) to see if you can make that assumptin. The good news is that this type of 'propensity model' is valid and can be used as a cheap substitute for retraining. But sounds like you know what you are doing so press on! $\endgroup$ Commented Apr 28, 2022 at 18:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.