I wanna ask very important question about the random population generation gin splitting the dataset in machine learning classification models;

For example to more explain, i used seed =1 a,d i got accuracy of 0.7 and seed= 5 and i got accuracy of 0.8 and seed= 2000 and i got accuracy of 0.89 using Adaboost.

I found research paper using the same datase i used and accuracy achieved is 0.94 using xgboost model without specifiying the seed used in developing the model.

same for other research papers exists.

My question is which results i ave to pick to compare my model with other models proposed in literature Meanwhile i implemented all the models proposed in literature with the different seed i used and i found not the same results in their paper and sometimes with not all the seeds my result with adaboost is better.

I need help to compare my proposal with other works

  • $\begingroup$ If you want to compare metrics, you need to compare distributions. As many methods are non-deterministic, it doesn't make sense to compare single values. You could run same method with single seed and still get different results. Repeat your experiment at least 50 times to make statistics work. $\endgroup$ – Piotr Rarus - Reinstate Monica Dec 3 '19 at 12:14

I agree that sometimes the authors are not that clear about some details. If a dataset is already splitted into train/test/validation, then there is not much to do (supposing the dataset is well made). Given a dataset not already splitted then you have (at least) two ways to test your model:

  • fix a random seed, check that the split is not biased and run several trainings. Even with the same split, the same model can converge to different local minima, because of the stochastic descent;
  • don't fix a random seed and run several trainings (but store the seed where you keep your experiment data!).

If your findings are not coherent with the literature, and you are sure there aren't bugs in the code, then you should ask specific questions or write to the authors. Depending on the model, you might try some standard validation techniques.

  • $\begingroup$ Thanks for you reply the dataset is not splitted. Fixing a seed, i have split data into train/test/validation and using cross validation to find the best hyperparameters of the model and then test the model on the test set to ensure a tradeoff between bias and variance. Each time i change the seed i found a result sometimes better than the results published in the literature sometimes worse. $\endgroup$ – Rawia Hemdan Jul 6 '19 at 9:05
  • $\begingroup$ how could i compare my results to literature? $\endgroup$ – Rawia Hemdan Jul 6 '19 at 9:06
  • $\begingroup$ Mainly i am not using neural network i am using stacking with some weak classifiers like DT, ... $\endgroup$ – Rawia Hemdan Jul 6 '19 at 9:13
  • $\begingroup$ @RawiaHemdan If the authors are not clear about how they produced their accuracy then you can't do much. Maybe they are considering an average accuracy. $\endgroup$ – dcolazin Jul 6 '19 at 11:31
  • $\begingroup$ Thanks @dcolazin i tried to use the mean also but for one run it takes one day so i f i take 100 runs it will take 100 days with a machine of 8GB RAM $\endgroup$ – Rawia Hemdan Jul 6 '19 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.