0
$\begingroup$

As a part of my master's thesis, I am using different ML models for prediction and classification. The problem is I am confused if I should use only the result for a fixed random_state(suppose 10) or use a different random_state each time. (for example, use 3 different random_state and take the mean of the result).

New contributor
nigebLM is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

1 Answer 1

0
$\begingroup$

Yes, ideally, you should run experiments with different random seeds.

The reason why it is recommended to use a fixed random seed is reproducibility, i.e. you don`t want to get different results every time you train a model. However, fixing the random seed does not solve the problem that results of any non-deterministic model will depend on the chosen random seed. It only ensures that you (or in this case, also your thesis supervisor) is able to reproduce the results.

But as the authors of Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approaches write:

[...] there is a high risk that a statistical significance in this type of evaluation is not due to a superior learning approach. Instead, there is a high risk that the difference is due to chance

because

Non-deterministic approaches like neural networks can produce models with varying performances and comparing performances based on single models does not allow drawing conclusions about the underlying learning approaches.

and therefore to

[...] not submit only a single model, but multiple models trained with different random seed values. Those submissions should not be treated individually. Instead the mean and the standard deviation of test scores should be reported.

This does not apply if you perform k-fold cross-validation since the random numbers generator will progress from fold to fold and, therefore, models in each fold be based on different random numbers.

Having said that, I would also check what validation strategies have been taught in your MSc specifically and what your thesis supervisor(s) think of this (e.g. check papers they have published). Moreover, since this is a master thesis for which you have limited time available you might need to be pragmatic too. If your model takes a week to be trained and there is a larger number of models to be trained then you might need to cut down on the number of experiments per model. If that is the case, I would highlight this as a limitation in your thesis.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you so much! $\endgroup$
    – nigebLM
    yesterday

Your Answer

nigebLM is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.