Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Working on a medical diagnostic convolutional neural networking problem, and it's not obvious (to me) how the dataset should be split up.
Do I have enough data to split it in 3, or should I just have train and validation?
What proportion of images should I put in each?
(research article links appreciated :])
For the problem of an imbalanced dataset, you can look into stratified sampling, or stratified-cross-validation (as mentioned here). One idea might be to create stratified batches from the data.
I would probably make all attempts to get train/val/test splits, because you will otherwise face issues when claiming a final test accuracy, as the model might well have seen your entire dataset.
One could imagine splitting to have e.g. 300/9750 (pos/neg) in the training dataset, and during training, you create stratified batches from those 1050 images, so each batch e.g. of 50 images, might contain, 10 positives and 40 negatives. This is still somewhat imbalanced, but you are pushing the balance into a more favourable direction in that the model should be able to learn more efficiently.
In medical research it is often the case that there are too few samples (in addition to class imabalnces), and so there is usually a huge effort that goes into data augmentation, which you might also be able to make use of it. Here is some related literature (extremely fresh - edited one week ago!).
Here is another approach, whereby the authors (William Fithian & Trevor Hastie) devise a subsampling method, which uses the features of the samples to accept/reject them. They design it for the simplist case (logistic regression), but perhaps it might give you ideas:
... using a pilot estimate to preferentially select examples whose responses are conditionally rare given their features.
Something to be especially aware of when using the ideas I mentioned above is overfitting. Cross-validation is probably what can best help you out in this respect.
Either you can resample the lower class and up their number upto the higher class or while applying train_test_split use stratified sampling so that it make sure that in your training samples you get both the classes.
There is one more way, which is applying weight on the correct detection of the lower class, which means that for each right detection of lower class will carry more weight that the correct detection of higher class, this can be applied in the loss funtion during optimization.
Simple thing to do is to use Stratified sampling as suggested by @n1k31t4. Other thing which people usually do with images is to Image Augmentation. So, you can try to rotate,tilt, mirror your positive data set such that it can increase to 13k. You can take a look about it Here
