1
$\begingroup$

I have searched in several web pages how to choose the best machine learning model for a dataset and they all seem to agree that they should be compared using the same seed. However, they only run the test once and choose the best result of all.

Shouldn't you run a lot of tests and average the results before choosing one? It may happen that for a given seed a model has obtained very good results when, in general, it does not give good results.

$\endgroup$

1 Answer 1

0
$\begingroup$

Indeed! If you were to re-train a given model repeatedly using different conditions you would generally find that the accuracy (or metric of your choice) forms a (probably) Gaussian. A single model is like taking one sample out of this distribution.

And the model’s initial conditions are not the only source of variance from this seed. Consider the train/test split of your data: exactly which 80% (or whatever) went to training vs test? The smaller the dataset, the more this matters.

$\endgroup$
3
  • $\begingroup$ Thanks for your answer, I found it really strange that it was not mentioned in the pages I consulted and I did not know if there was a reason behind that. And yes, I also assumed that how the division of the data is done is a fact that influences the results produced by the model. I will do several tests to make sure I choose the best models, thank you very much! $\endgroup$
    – anmarlea
    Aug 13, 2021 at 12:04
  • $\begingroup$ I suppose it’s an inconvenient thing to consider as it requires many training cycles. Depending on the data and model, this could be really daunting. Might be a good practical tradeoff to just train once and take your chances… after all, on average you’ll get an average measurement of an average model. :-) $\endgroup$
    – G__
    Aug 14, 2021 at 0:13
  • $\begingroup$ Don't worry, luckily it is a small data set so there is nothing wrong with doing several tests! $\endgroup$
    – anmarlea
    Aug 14, 2021 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.