1
$\begingroup$

When training the model, I understand

  1. If I supply too many on a certain category, it may become overfitted and treat almost all predictions as the overfitted category. This can lead to false positives in the overfitted directory.
  2. If instead, I supply the least amount of photos to equally distribute, is the probability of of a rarer category appearing is now equal to a category that would be common to find? This can lead to false positives in the rare category but may turn my false positives in #1 to be true negatives.
  3. If instead I skip training the rare category, I get false positives in other categories.

How do we account for the natural distribution of our target classifications (for example, types of common lesion vs rare lesions)?

Is it best practice to equally distribute? If so, what should we do if we have so little of the rare category, the sample images are orders of magnitude smaller?

$\endgroup$
  • $\begingroup$ The bold font is hurting my eye.. $\endgroup$ – DuttaA Jul 6 '18 at 14:57
0
$\begingroup$

About equalizing the number of samples for each class, it is not desirable at all. The distribution of each class should be real. There are numerous reasons for that, but the main point is that the distribution of your train and validation data should be like the distribution of your test data. Take a look at here.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I think I understand, but can you point me to the numerous reasons so I may learn about them? What you're saying is, I should I keep the distribution "real", meaning in the order of expected real world occurrence? Like say, more occurrences of benign moles rather than malignant cancerous melanoma so that the classifier has more tendency to have true positives in natural real world distribution? $\endgroup$ – Azeworai Jul 7 '18 at 0:03
  • $\begingroup$ No, not the world. The region or the environment that you're going to use your classifier and test it in that environment. The data should be the real nature of the environment which your model is going to be deployed there. Suppose you want a classifier to distinguish whether an input feature pattern is male or not and your feature vector consists of height. In eastern countries 175 with high probability can express that the input is referring to a male but in western countries you cannot say with confidence that it is a male due to the fact that women are taller in those countries! $\endgroup$ – Media Jul 7 '18 at 1:19
0
$\begingroup$

Depending on the method that you use, this answer might be different.

First, you split the data between training, validation, and test splits. Forget about the test set. Keep the validation data in its natural distribution of classes.

You have several choices with the training set data -

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.