2
$\begingroup$

I am using Convolutional Neural Networks (CNN) and I just want to ask if the way I split my training/validation/testing set is correct.

I have a total of 55 subjects. I plan to split them into 80–10–10 for training (45 subjects), validation (unseen 5 subjects), testing (unseen 6 subjects).

Should the validation set consists of unseen subjects as well? Or can I shuffle the whole training set and use a part of it (10-20%) as validation set?

I have read that using N-Fold cross-validation, the whole training set (instances) are shuffled then split into N-folds and the model is trained and averaged N times. However, in the case for Neural Networks or CNN, we don't use cross-validation since it is very computationally expensive.

I'm just wondering which is correct since using a validation set of unseen subjects, my model starts to overfit after 3-5 epochs and doesn't learn at all. On the other hand, if I use 10-20% of the training set as my validation set, my model learns with reasonable accuracy (45-50%) using a 3-layer CNN but when tested with the unseen testing set, my top-1 accuracy is around 15-16% only.

Thank you very much.

$\endgroup$
  • $\begingroup$ How are you validating with unseen subjects? $\endgroup$ – Hima Varsha Oct 4 '16 at 7:35
  • $\begingroup$ @HimaVarsha: OP likely means "unseen during training", and is splitting fully labelled data to train/cv/test. If all images of one subject are not in training set, but are in test set, then that user is "unseen" as far as training the network is concerned. That means the test results will be a good indication of performance against completely new subjects. $\endgroup$ – Neil Slater Dec 3 '16 at 9:41
3
$\begingroup$

NB This advice assumes your goal is to recognise expression in pictures of any person, and not just people from your training data.

Should the validation set consists of unseen subjects as well?

Yes. This will give you the most accurate measure of performance in the task you want to use the network in, in order to choose the best generalisation and take it forward to testing.

You would only use a simple random split if the end goal of your trained network is to recognise expressions from images of the people in the training set.

Or can I shuffle the whole training set and use a part of it (10-20%) as validation set?

No. If you take random samples where the same face appears in training and cv, you will get an over-estimate of generalisation. I have seen this effect first hand in the Kaggle State Farm Distracted Driver contest, and you should see it discussed in the forums there. It might be helpful for ideas to improve performance too.

if I use 10-20% of the training set as my validation set, my model learns with reasonable accuracy (45-50%) using a 3-layer CNN

This is a data leak between train and cross validation - the network has learnt to correctly classify expressions in images of people it has already seen, and that is what you are measuring by taking a split in this way. It is not surprising that the test result does not match the promising values from cross validation.

I'm just wondering which is correct since using a validation set of unseen subjects, my model starts to overfit after 3-5 epochs and doesn't learn at all

You are over-fitting regardless of how you split train,CV. When you split the CV incorrectly, then in addition to over-fitting, the data leak gives you bad guidance.

It looks like that you have very little diversity in the training data, whilst training an image classifier from scratch requires a lot of data. Consider:

  • Collecting more labelled data, perhaps some other dataset you can download.

  • Adding a lot of regularisation to your model (e.g. multiple dropout layers)

  • Using a pre-trained image classifier network (e.g. VGG-19 or Inception) as a starting point and fine-tuning it for your classification task.

  • Use full k-fold cross-validation regardless of the computational cost, to mitigate problems of using small training set. This won't help solve your training problems directly, but will give you a better shot at tuning your network hyper-parameters once you solve that issue.

$\endgroup$
2
$\begingroup$

Validation set implies it is obtained by taking a part of training dataset. It cannot be unseen(I am assuming by unseen, you mean whose labels you do not know or you want to check on) subjects because then we will not be able to validate(check accuracy/precision to test our model) in the first place. I think what you are doing is probably taking validation set from training dataset which you are using for training as well leading to overfitting. Let me be elaborate:

Let us assume you have 100 documents. Use 80 for training and rest 20 for validation. DO NOT use these 20 documents for training. If you use it, you will lead to overfitting(I think this is probably the answer for your overfitting). Now, the other test dataset(unseen documents) can be tested on.

$\endgroup$
  • $\begingroup$ Thank you for your answer but sorry, I don't think we're on the same track. What I meant by unseen was subjects (faces) that the model have never seen before and it has labels. So, whole data set consists of 55 subjects, 80% training (45 subjects) then 20% validation (10 subjects). 1.) Would it be correct to validate the model on unseen faces and instances 2.) Or given the whole data set (55 subjects) I shuffle all the instances, then take 20% of it as my validation set. So it means that faces of subjects inside the training set can also be seen in validation but not the same instances. $\endgroup$ – Renz Oct 5 '16 at 1:26
  • $\begingroup$ @Renz, shuffle the whole dataset and then choose the validation set of 20%. This is the normal practise. This enables us to validate on random dataset which will help for better evaluation. So, I'd prefer the second way. $\endgroup$ – Hima Varsha Oct 5 '16 at 4:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.