1
$\begingroup$

I am training a model for 3D CT scan classification (is there a nodule or not in the image).

I'm using a 3D CNN and I obtaint the following curve;

enter image description here

(blue curve is the validation data (10% of dataset))

as you can see, I obtain good result on both validation loss (categorical cross entropy, up to 0.16 for val) and metric (categorical accuracy) (up to 1.0 for val) but not on the training data.

I'm using a grid deformation on the training data and several dropout layer since my dataset is very small (120 samples), so it could explain the lower result on training set.

Still, I dont know if I can accept this type of result ?

Improve this question
$\endgroup$
4
  • $\begingroup$ How big is your data in total, could 10% for validation be to small? It concerns me a bit that your training performance is almost flat (no improvement) while the validation curve fluctuates wildly. $\endgroup$
    – Fnguyen
    Jul 17, 2020 at 9:11
  • $\begingroup$ as stated in my post, my dataset is as small as 120 samples. Thats why I'm using on the fly data augmentation technique. Of course such a small dataset is a problem, but I can't obtain more data and have to deal with it. The fact that the training data are augmented on the fly could explain the lack of improvement on the training curve (which is also my main concern) $\endgroup$
    – Chopin
    Jul 17, 2020 at 9:17
  • $\begingroup$ Quite honestly I think a validation set of 12(!) is more concerning to me, this means validation performance is almost random. I would try augmenting data before the split and/or increasing the size of the validation data. I think your problem is less that the training performance isn't improving but that the validation score isn't trustworthy. $\endgroup$
    – Fnguyen
    Jul 17, 2020 at 9:21
  • 1
    $\begingroup$ You can try using cross-validation in order to use all your data and get a more reliable validation score. Yet, with so few data, I doubt that a CNN will be of much help to solve your problem $\endgroup$
    – qmeeus
    Jul 17, 2020 at 9:59

2 Answers 2

Reset to default
1
$\begingroup$

The validation set having just 12 samples is too small to consider deploying a model which performs well on the validation set.

A model that performs well on the training set with 110 samples and then performs better on the small validation set would be preferable.

Since your set is too small, all though it is not prudent, you should consider running the model on the entire 120 samples to see how it performs on the entire dataset.

I presume you can't get more samples, if you can, that would be a good step.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Leave-out-out cross-validation is an approach to consider, too. $\endgroup$
    – Dave
    Jul 17, 2020 at 10:30
1
$\begingroup$

Use of the test set is to keep us away from overfitting the model on all the available data

So, we do all our training/tuning on a set of data and then test it on an unknown set.
It gives us the confidence that the model is generalized(though it may still fail on new data) but this is the best we may do.
It never means that a good score on the test set is great even if I know it has not learned the train set properly
It is happening(most likely) because your test set is very small

As an analogy, let's assume that your train set has 20 different variances (in very simple terms), your model learned ~14 of them.
Your test set has only 5 variances out of which 4 is from the 14. So, it will give a good score but you should expect that in the real scenario, the model will fail on 7 out of 21(33%).

- Try with a bigger test set and a smaller train set(70/30) and see what happens
- Try K-Fold with small K values i.e. 3/5

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.