I am wondering over whether the number of classes distributed over my training, validation, and test label affects the model.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Your question is a little misleading. There are two potential issues (1) class imbalance in training data (which is the case here); (2) class imbalance between training and testing (not the issue here since the proportions of class labels seems balanced). Also differentiate between 'number' of labels and 'proportion' of labels, which, although related, are nonetheless different challenges. You may want to check out literature on extremely rare data/event classification methods. This survey paper may help: journalofbigdata.springeropen.com/articles/10.1186/… $\endgroup$– Dynamic StardustMar 5, 2021 at 18:42
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
yes of course, the model will learn to recognize the label 0.0 very well and maybe have an idea about the others, unbalanced dataset is common problem in ML, look at Oversampling and Undersampling
$\begingroup$
$\endgroup$
When carrying machine learning, having a (even quick) look at distribution within the data set helps in getting a little understanding of how data has been generated. Most of the time, and for a broad number of professional projects, since you have to carry on with the data you've got, you cannot make a lot of assumptions from this kind of observations.
When it comes to distribution between training, validation and test sets, it's kinda slightly different though. Indeed, it's very recommended to at least have a quick glance at how data is distributed within the three sets, and even ensure that this distribution is the same. Put simply, if you build a model using an algorithm and associated parameters using your training set, it seems reasonable to argue that such model won't be able to describe what's going on within a data set sourced from other processes. I usually think of model parameters as characteristics of the data generating process that gave birth to the data set I have to work with. Not willing to reinvent the wheel here, so I'd point to this answer which explains it quite well, using a toy example.
Concerning oversampling and undersampling as mentioned above, these are pretty useful methods of ensuring that an imbalanced distribution between target labels in your data set would be properly extended to the process of splitting it into training, validation and test sets, while avoiding bias propagation due to the difference in occurrences between labels.
