0
$\begingroup$

Which is the correct way - presenting the results of the best or the last iteration on the dev set in a paper? In research papers I usually see only one value, is it the best iteration of all?

I'm asking because my model training last iteration wasn't the best one. Instead the best iteration was somewhere in the middle of the training.

$\endgroup$
1
$\begingroup$

Is dev set a validation set here? Your best model occurred where the error on the validation set was lowest. That's the model you choose. However the validation error will be a slightly optimistic estimate of the true generalization error. Ideally you have held out a test set as well, and evaluate and report error on that.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Yes, the dev set is a validation set in this case. So if I want to show results on both dev and test set, I can publish the best iteration for each set, e.g. 100th iteration on dev set and 150th iteration on test set? This makes sense to me, just wanted to be sure that I'm not breaking some unwritten conventions. $\endgroup$ – T.Poe Jan 16 at 14:48
  • $\begingroup$ No, you choose your model based on the validation set. You estimate its generalization error on the test set then, though. You are not evaluating the test set during training at all, only the validation set. $\endgroup$ – Sean Owen Jan 20 at 14:39
0
$\begingroup$

I agree with @Sean Owen's above answer. Best Result or best accuracy cant be decided from the validation set because sometimes there may be a possibility that datapoints come in validation set has been seen during training so obviously it will have good accuracy score. Best way to calculate benchmark on test or holdout set. Another way might be one can take an average of all cross-validation score and see model progress.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.