1
$\begingroup$

I'm trying to develop my neural network with both early stopping and bayesian regularization (matlab implementation, lm algorithm is used for both). Since in bayesian regularization I have not the validation set, how can I compare the generalization capability of the networks obtained with the two methodologies? Thanks

$\endgroup$
4
  • 1
    $\begingroup$ Why don't you have a validation set? Is it just because the implementation doesn't create one for you? Not knowing a lot about bayesian regularisation, my naive answer would be "so set aside a validation or test set manually" . . . in fact, as long as you are not tuning your params using the set, but performing a single pre-decided comparison of generalisation success, then I would call that a "test set" $\endgroup$ – Neil Slater Jan 4 '15 at 14:36
  • $\begingroup$ maybe I can compare the validation mse(with the name 'validation' because it is used for parameters tuning) of the trainlm with the test mse (called test because the paramerets are tuned in another way)? $\endgroup$ – Daniel Jan 4 '15 at 16:34
  • $\begingroup$ @NeilSlater I forgot to tag $\endgroup$ – Daniel Jan 4 '15 at 17:33
  • $\begingroup$ I would not know how to assess whether it was safe to compare those two metrics across the types of NN training you are using, maybe you should wait for a full answer. However, I'd just hold out a test set, not used to train or optimise params for either network, and compare results from both networks on that. If you can afford to keep enough labeled data aside, this is the simplest and least susceptible to errors in assessment. $\endgroup$ – Neil Slater Jan 4 '15 at 18:41
2
$\begingroup$

As Neil said in the comments, split out a test set (data you don't use in training either model) and see how each trained model performs on the test set.

$\endgroup$
1
  • 1
    $\begingroup$ ideally also ensure the test set is properly separate from the training set. Eg if doing handwriting recognition have the test set have writing done by people who's writing (on other letters) is not in the training set. $\endgroup$ – Lyndon White Jan 8 '15 at 1:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.