I would like to ask a question about the random population generation gin splitting the dataset in machine learning classification models.

For example, I used seed = 1 and got accuracy of 0.7 and seed = 5 and got accuracy of 0.8 and seed= 2000 and got accuracy of 0.89 using Adaboost.

I found a research paper using the same dataset i used and accuracy achieved is 0.94 using xgboost model without specifying the seed used in developing the model.

Same exists for other research papers as well.

Which results I need to pick to compare my model with other models proposed in the literature? Meanwhile I implemented all the models proposed in literature with the different seed and found that results are different when compared to the paper. And sometimes without using my result with adaboost is better.

I need help to compare my proposal with other works.

  • 1
    $\begingroup$ If you want to compare metrics, you need to compare distributions. As many methods are non-deterministic, it doesn't make sense to compare single values. You could run same method with single seed and still get different results. Repeat your experiment at least 50 times to make statistics work. $\endgroup$ Dec 3 '19 at 12:14

I agree that sometimes the authors are not that clear about some details. If a dataset is already splitted into train/test/validation, then there is not much to do (supposing the dataset is well made). Given a dataset not already splitted then you have (at least) two ways to test your model:

  • fix a random seed, check that the split is not biased and run several trainings. Even with the same split, the same model can converge to different local minima, because of the stochastic descent;
  • don't fix a random seed and run several trainings (but store the seed where you keep your experiment data!).

If your findings are not coherent with the literature, and you are sure there aren't bugs in the code, then you should ask specific questions or write to the authors. Depending on the model, you might try some standard validation techniques.

  • $\begingroup$ Thanks for you reply the dataset is not splitted. Fixing a seed, i have split data into train/test/validation and using cross validation to find the best hyperparameters of the model and then test the model on the test set to ensure a tradeoff between bias and variance. Each time i change the seed i found a result sometimes better than the results published in the literature sometimes worse. $\endgroup$ Jul 6 '19 at 9:05
  • $\begingroup$ how could i compare my results to literature? $\endgroup$ Jul 6 '19 at 9:06
  • $\begingroup$ Mainly i am not using neural network i am using stacking with some weak classifiers like DT, ... $\endgroup$ Jul 6 '19 at 9:13
  • $\begingroup$ @RawiaHemdan If the authors are not clear about how they produced their accuracy then you can't do much. Maybe they are considering an average accuracy. $\endgroup$
    – dcolazin
    Jul 6 '19 at 11:31
  • $\begingroup$ Thanks @dcolazin i tried to use the mean also but for one run it takes one day so i f i take 100 runs it will take 100 days with a machine of 8GB RAM $\endgroup$ Jul 6 '19 at 14:09

The best way to avoid your issue is using a K-Fold Cross Validation. Having a 0.1 difference in accuracy between 2 seeds is a lot, so I'd suggest you doing cross validation, with a random splits (you can, for example, shuffle your data before entering cross-validation loop). Running 2 times your Cross-Validation Loop, shuffling data before, could give you a bit of a change (because of the shuffle, cross validation sets won't be exactly the same), but I highly doubt it'll be as high as 0.1 Then you could compare your model, since you're using a stable evaluation process.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.