2
$\begingroup$

for an object detection task I created a small dataset to train an object detector. The class frequency is more or less balanced, however I defined some additional attributes with environmental conditions for each image, which results in a rather uneven distribution of classes depending on the viewed attribute (e.g. class X occurring with attributes A and B).

I used a typical training/validation split and the loss curve lets me conclude that no over-fitting is occurring. I am aware that in general for testing unseen images with a realistic feature distribution should be used, however due to the small size of the dataset, splitting off additional 10 or 20 % for testing or randomly drawing examples would result in certain combinations of classes and attributes not or only barely occurring in the test dataset and manually selecting examples would be very time-consuming.

I am therefore now wondering, if it reasonable to use the whole dataset (train+val) for testing and calculation of performance metrics as a compromise since I see no signs over-fitting?

Improve this question
$\endgroup$
2
$\begingroup$

Considering that you have not used a cross-validation strategy, you could try to use the LOOCV (Leave One Out Cross Validation) strategy, so you have several splits (as many as samples considered in your training dataset) and still leave some samples as a final never-seen-before validation set.

This is costly from a performance perspective, but interesting for small datasets as in your case. Another option, in case the dataset is not really small, is usin k-fold cross-validation (with k being 5, 10...).

A post with practical info can be found here and a related topic discussion can be found here

Improve this answer
$\endgroup$
3
  • $\begingroup$ Mmm that would require to train N models if I have N images right? Since I have 4500 images, this is out of a computational feasible range for me. But maybe a 5-fold cross validation could be sufficient. $\endgroup$ – P. Leibner Jan 20 at 15:15
  • 1
    $\begingroup$ of course, k-fold cross-validation is the best option (k=5 or 10 usually), I thought your dataset was even smaller $\endgroup$ – German C M Jan 20 at 16:51
  • 1
    $\begingroup$ Yea, I should have added that info earlier. I will enlarge my validation set and use cross validation and rather sacrifice on performance instead scientific principles. Thanks! $\endgroup$ – P. Leibner Jan 20 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.