I have a question related to performance variability and how to assess different methods. I want to compare the result of 5 different classifiers on the same dataset (let's say 20 newsgroup dataset).

I have been asked to assess the performance variability with random sample variation, and then report the statistical significance.

What I had done was that, I used train_test_split and run each model 10 times and then report the average of the f1 score.

I know there are cross-validation and other resampling methods. but I cannot re-run on all 5 models with the time I have.

So my first question is that can I explain to them scientifically the approach I followed is ok and it gives a good estimation of the fact that our model performance is not achieved by luck?

And the second question is that with the approach I followed is there any way I can calculate the significance importance?

Appreciate any input on this.

  • $\begingroup$ Any thought on this :( $\endgroup$
    – Maria
    Aug 24 at 4:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.