1
$\begingroup$

I'm building a Feedforward Neural Network with Pytorch and doing hyperparameters tuning using Ray Tune. I have train, validation and test set, train and validation used during the training procedure. I have different versions of the model (different learning rates and numbers of neurons in the hidden layers) and I have to choose the best model. But I'm unsure on which metric should I use to choose the best model. Basically I don't know the XXXX in this line of code:

analysis.get_best_config(metric='XXXX', mode='min')

Should it be test loss or validation loss?

Improve this question
$\endgroup$
2
$\begingroup$

Essentially, the function of your testset is to evaluate the performance of your model on new data. It mimics the situation of your model being put into production. The validation set is used for optimizing your algorithm.

Personally I would recommend tuning your algorithm using your validation set and using the hyperparameters of the training epoch with the lowest validation loss. The accuracy of the model using these hyperparameters on your testset can be used to estimate the performance of your model in the 'real world'.

Improve this answer
$\endgroup$
  • $\begingroup$ Thanks! So testset used only when I have the completed model with all his hyperparameters defined $\endgroup$ – fabio_vac Nov 17 '20 at 9:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.