2
$\begingroup$

I am thinking about the prospective application of a trained classifier in a real-world context. We know that when we do over/under-sampling to balance our dataset, we never touch the testing set as we want to keep our dataset's real behaviour. But the part that I do not understand is the role of the test set's distribution in a classifier's performance.

Let's say I have a model that can label an email as spam or non-spam. If I launch this model in my email-service, in a specific time window, all the emails that my classifier receives might be non-spam. But the trained model has a 50-50% distribution for each category. My question is, does this difference in the distribution--during the prospective application-- change the performance of the model? e.g. if my web-service receives 5 spams and 5 non-spams in that time window, should I receive a more accurate classification? Based on my understanding, the answer should be a No. Still, I see everywhere that people are talking about the importance of the test distribution and its role in the performance and accuracy of predictive models.

Thank you.

$\endgroup$
3
  • $\begingroup$ datascience.stackexchange.com/questions/17873/… $\endgroup$ – Ethan Nov 16 '20 at 2:34
  • $\begingroup$ Thank you @Ethan. However, the question you referred to is addressing the generalizability problem, which is the main reason for doing over/under-sampling. My question is more about the distribution of the testing set. In plain English: should we be concerned about this distribution at all? $\endgroup$ – Sarah Nov 16 '20 at 2:45
  • 1
    $\begingroup$ Your question isn't very clear but your test distribution should mimic what you expect to see in the future. The idea of the test set is to estimate how well your model will do on future data, if your future data has a different distribution than your test set than your estimate you get from the test set won't be very good will it. $\endgroup$ – astel Nov 16 '20 at 6:55
1
$\begingroup$

Depends if you are going to do online learning.

Lets say you will do online learning/incremental learning than test set Distribution will make difference. For example because of catastrophic forgetting of neural Networks.

If you are making Batch predictions than it makes no difference whats the test set Distribution. Model knows no difference since it does not Change ist state.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.