I have a fixed amount of dataset, say text data with 100,000 records, to rank classifiers, such as gradient boosting, random forest, and other ones, according to their classification performance. The dataset is divided randomly for training and validation but the issue is that I get different prediction performance even if I use the same dataset overtime.
My understanding is that once the classifiers are ranked after the first training and validation run on the dataset, the rank should always show the same way as the previous rank. I don't add or delete the fixed amount of dataset that I use. Even though the same dataset is divided randomly, what I think is there shouldn't be a significant impact on the performance because the ratio of data used between training and validation is 80(training):20(validation). If they change their behaviors each time I run on the same unchanged fixed amount of data, how can I tell which one shows better performance on the current dataset? I understand the training data changes but its percentage is 80% of the dataset.