1
$\begingroup$

Assume we develop a model for a binary classification task that reaches a certain Gini/AUROC estimate on the validation ( or training ) sample, among others. This is an overall good metric, often used for evaluating the ability of the model to separate the samples into, say, goods vs bads.

Further, assume this model is adequate and will be used for further collection of new samples with a certain cutoff value. What should be expected Gini/AUC estimates on the newly collected sample?

From what I'm noticing, on the training sample there were clear cases where the model was able to distinguish and separate it with large probabilities. On the other hand, with applied cuttoff, say, <50%, the new sample with collect only those cases where no such clear separation is possible (because if it would, the case might not get collected). With such approach, for me it seems logical that the overall separation in the new sample will be lower, resulting in lower out-of-development-period Gini/AUC.

Is this the expected behaviour in normal production environments? Am I understanding things correctly?

Note: I understand that there are other simple metrics, such as sensitivity/specificity, hoslem.test and others, allowing for measuring and visualising True/False Positives. However, I have found that Gini/AUC is often a key metric when discussing and comparing classification models.

$\endgroup$
1
$\begingroup$

The advantage which train/test/validation dataset separation has is that you separate your dataset into:

  • The individuals which you know the exogenous variables and the output: Training
  • The individuals which you know the exogenous variables and the output (but you suppose you don't know which the output is): Test
  • The individuals you know the exogenous variables but not the output: Validation

Every DS or ML model is made so it is prepared to receive a validation dataset in the future and try to get every metric just almost as good as if it was the train dataset.

The test dataset has the objective of simulating the situation of having data but not output, and then you have the output to measure the behaviour and comparing the modelled vs real output.

So, for a concrete answer: The behaviour you should expect from the validation (or newly collected sample) is the same as the test dataset.

Given that the underlying phenomenon and sampling technique remains the same.

For more information: https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7

$\endgroup$
9
  • $\begingroup$ Thanks. My main question in a way is about AUC as a metric, which is a full sample based metric. Naturally, its properties should change when the validation sample is censored (applied cutoff rule), but how much change is expected? I understand there are other metrics, but gini/auc are too popular to ignore :) $\endgroup$
    – runr
    Apr 11 '19 at 21:16
  • $\begingroup$ In other words, after applying cutoff, as in your bolded sentence, the sampling technique will not be the same anymore. So, whats then? $\endgroup$
    – runr
    Apr 11 '19 at 21:17
  • 1
    $\begingroup$ There is no metric which is more or less sensitive to overfitting than others, so from AUC/Gini you should expect the same: You should expect the same decreased as when comparing test vs train datasets. $\endgroup$ Apr 11 '19 at 21:18
  • 1
    $\begingroup$ You could make decisions on the cutted-off sample, given that you cut them based on a previous good model. What you are doing when you cut them off is giving bias to your sample, but you are not changing the underlying phenomenon. You are safe if you do not run models on the new data. Your model will be out of date some day. Analysing what happens with the AUC, you will face a decline, because you will keep all the >50% individuals, and is more difficult to distinguish between better-and-good than to distinguish between good-and-bad. Please let me know if my answer was useful $\endgroup$ Apr 11 '19 at 21:50
  • 1
    $\begingroup$ Juan's last comment is really the answer here: assuming the test set is really representative, you can just apply a 'previous good model' score the resulting subset with the new model (mimicking what you'll be doing later, as you note in your last comment's last sentence). And toward that last question, I'm not aware of anything very satisfactory, but search for "rejection bias" and "reject inference". $\endgroup$ Jan 14 '20 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.