Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Which is the correct way - presenting the results of the best or the last iteration on the dev set in a paper? In research papers I usually see only one value, is it the best iteration of all?
I'm asking because my model training last iteration wasn't the best one. Instead the best iteration was somewhere in the middle of the training.
Is dev set a validation set here? Your best model occurred where the error on the validation set was lowest. That's the model you choose. However the validation error will be a slightly optimistic estimate of the true generalization error. Ideally you have held out a test set as well, and evaluate and report error on that.
I agree with @Sean Owen's above answer. Best Result or best accuracy cant be decided from the validation set because sometimes there may be a possibility that datapoints come in validation set has been seen during training so obviously it will have good accuracy score. Best way to calculate benchmark on test or holdout set. Another way might be one can take an average of all cross-validation score and see model progress.
