I have a question related to performance variability and how to assess different methods. I want to compare the result of 5 different classifiers on the same dataset (let's say 20 newsgroup dataset).
I have been asked to assess the performance variability with random sample variation, and then report the statistical significance.
What I had done was that, I used
train_test_split and run each model
10 times and then report the average of the
I know there are cross-validation and other resampling methods. but I cannot re-run on all 5 models with the time I have.
So my first question is that can I explain to them scientifically the approach I followed is ok and it gives a good estimation of the fact that our model performance is not achieved by luck?
And the second question is that with the approach I followed is there any way I can calculate the significance importance?
Appreciate any input on this.