I'm training a neural network with a very small dataset just to get things set up, before training on a much larger set. (I only have about 500 data points available to me at this time, with more coming over the weekend.) I should also mention that I'm exploring whether or not this data has enough of a signal to even warrant trying to train a model on it.

I'm currently working with ~500 feature vectors with 4 features each.

I find that when I have a training set containing about 90% of my samples, randomized, that it typically performs significantly worse than the small set of validation samples being validated on that modal. I would expect the opposite.

Edit: I should mention that I'm using RMSE for my loss function

What could this mean?

Some thoughts I have are that I just need more data points to get a good read from the validation set. But I'm also wondering if it could mean that there just isn't a good signal in the features I'm using. But if that was the case, wouldn't my model overfit to the random-ish data and then still perform worse on the validation set?

  • $\begingroup$ What performance measure are you using? Are your train/test sets stratified by outcome (i.e. same class proportions in both)? Have you tried multiple train/test splits with the same result? $\endgroup$ – Nuclear Wang Jan 25 '19 at 14:54
  • $\begingroup$ @NuclearWang I completely forgot to mention that. Edited. Thanks $\endgroup$ – wheresmycookie Jan 25 '19 at 14:58
  • $\begingroup$ Do you still find this when you change the random number seed that selects the training and test sets ? $\endgroup$ – Dom Jan 25 '19 at 16:41
  • $\begingroup$ Maybe you have class imbalance in your training set $\endgroup$ – shadi Jan 26 '19 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.