2
$\begingroup$

My current advisor at Uni insists that I train 10 instances of the same network and pick the one with best test accuracy in order to escape the "local minima".

In my opinion this does not work at all, and should lead to picking the model that best fits the test_set, but may not be generalizable enough for it to work with the actual distribution behind it.

Is there any material or research on this? I really think this method is archaic and makes no sense, but i can't argue with my professor without actual scientific material.

Improve this question
$\endgroup$
5
  • $\begingroup$ What exactly is your objection? It sounds like you think finding the global minimum of the loss function is undesirable. $\endgroup$
    – Dave
    Commented Jul 10, 2020 at 23:15
  • $\begingroup$ My point is that retraining the same network 100 times over with different random starting weights and picking the one with best test_accuracy will not necessarily yield a better model than training just one. The one out of the 100 that has the highest accuracy in the test set might very well perform worse in real world distribution, because the test set is limited. $\endgroup$
    – Heitor da Rocha Coimbra
    Commented Jul 11, 2020 at 2:40
  • $\begingroup$ Off the top of my head, how would the average value of the weights compare to the weights of each trained model? I think picking one out of 10 is arbitrary, averaging 10 is also arbitrary, but is more representative of the work $\endgroup$
    – GK89
    Commented Jul 11, 2020 at 3:06
  • 3
    $\begingroup$ Why not just divide the dataset into training, validation, and test set? Anyway, never select a model by considering the performance on the test set. It is machine learning 101. $\endgroup$
    – sentence
    Commented Jul 11, 2020 at 10:05
  • $\begingroup$ By "10 instances of the same network and pick the one with best test accuracy" you mean 10 instances with the same hyperparameters? $\endgroup$
    – Jonathan
    Commented Jul 11, 2020 at 12:35

2 Answers 2

Reset to default
5
$\begingroup$

You are absolutely correct that this is an problematic approach. Your testing set should only be used at the last possible stage before deploying a model.

By using your testing set to make modeling decisions you will introduce bias which will favor the observations found in your test set and may not generalize. In an ideal world, your test set would represent the real distribution perfectly, however in practice this is never the case. Thus the suggested approach would result in a model that matches the test set's distribution more closely, but does not generalize to the real distribution.

The correct approach is to separate your data into a

  • training set,
  • validation set and,
  • testing set.

You then set the testing set aside until you have chosen a final model. With the validation set you are very much permitted to do what your professor suggested. You can fit multiple models and then pick the best one, or use all the models together in some bagging structure. Once you are satisfied you can then test against your testing set to see if the selected model generalized well.

Improve this answer
$\endgroup$
3
$\begingroup$

What your advisor is suggesting is called data leakage and it’s somewhat similar to training and testing your model on the same data. You might find it useful to read about p-hacking and backtest overfitting to get a feel for why this is a problem in quantitative finance. There’s also this excellent comic strip about the related concept of p-hacking...

The right thing to do is to train your ten (or more) models and then take the average performance as an indication of how well that type of model performs. You might also consider cross validation strategies to give you an even better idea of how your model might generalise.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.