Fair warning, I'm new to this field, so my process may be odd. Any advice is appreciated.

I am currently training a model to reproduce some DFT (density functional theory) data. I have been doing some feature selection, and decided on a set of features that seems to produce good results. I am now trying different initial randomizations of the same neural network structure to see if I can find one that happens to be particularly good. I tried 12 randomizations, all of which have the same training set, with a different, random training - validation split. The split is always 90% training, 10% validation.

One of these initial conditions resulted in a lower validation error than training error. The ratio of validation to training error was about 0.92. At first, I thought that this was just because of the randomization of the split between training and validation. I have now restarted the training, with several different learning rates, but with the same initial condition. The validation error is still lower than the training error. This is despite the fact that the training - validation split is being re-randomized.

My primary questions is whether or not this is a sign that I'm doing something wrong. If this is a legitimate result, then it's great.

Other info:

Fully connected feed forward neural network. Two hidden layers. No difference between training and validation model, they are identical in structure. I'm using PyTorch and the LBFGS optimizer. There is a reduction matrix at the end of my model that I can explain in more detail if anyone thinks it is relevant. All it does is sum different chunks of results together before comparison with a known value.

  • $\begingroup$ Quick update: Several of the training runs that I started off of the weird initial condition have finished. Those that got below a certain error no longer have a higher training error than validation. It appears as though the validation can reach a sort of minimum. $\endgroup$ – derangedhk417 Jul 15 '19 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.