I'm training a neural network with a very small dataset just to get things set up, before training on a much larger set. (I only have about 500 data points available to me at this time, with more coming over the weekend.) I should also mention that I'm exploring whether or not this data has enough of a signal to even warrant trying to train a model on it.
I'm currently working with ~500 feature vectors with 4 features each.
I find that when I have a training set containing about 90% of my samples, randomized, that it typically performs significantly worse than the small set of validation samples being validated on that modal. I would expect the opposite.
Edit: I should mention that I'm using RMSE for my loss function
What could this mean?
Some thoughts I have are that I just need more data points to get a good read from the validation set. But I'm also wondering if it could mean that there just isn't a good signal in the features I'm using. But if that was the case, wouldn't my model overfit to the random-ish data and then still perform worse on the validation set?