1
$\begingroup$

I am working on a classification problem in which ~90% of samples come from class 1 while ~10% of samples come from class 2. I have been using various techniques to combat the class imbalance while learning the problem, however, I am concerned about potential bias this may introduce because the true class distribution is unknown. Is it bad practice to weight classes during learning if the true distribution (or some reasonable approximation) is unknown?

$\endgroup$
2
$\begingroup$

Your assessment is right. You must first determine the data distribution in real-time (production) and only after that proceed with train_set, test_set and validation_set creation with the same distribution. And subsequently work on model training and setting the class weightages if required.

Why:

  • Any metrics upon which you evaluate your model are basically unreliable as it does not indicate the model's true performance (as on the true/real distribution).
  • You end up training and optimizing your model on a false perception of metrics.
  • Model weightages and bias depend on the training data's distribution. To get the best performance it must be trained on the true distribution.
$\endgroup$
2
  • $\begingroup$ Got it I figured that was the case. I suppose when the true distribution is unknown, we should weight classes (or resample) based on our best guess at the true distribution? $\endgroup$
    – tensormoby
    Nov 8 '21 at 15:52
  • 1
    $\begingroup$ Yes exactly, an approximation should be enough. Once you go live you can observe the actual distribution and adjust the weights accordingly. Just make sure you invest time in finding a close approximation so that you don't have to change it a lot in the future! $\endgroup$
    – Akshay
    Nov 8 '21 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.