0
$\begingroup$

I have a fixed amount of dataset, say text data with 100,000 records, to rank classifiers, such as gradient boosting, random forest, and other ones, according to their classification performance. The dataset is divided randomly for training and validation but the issue is that I get different prediction performance even if I use the same dataset overtime.

My understanding is that once the classifiers are ranked after the first training and validation run on the dataset, the rank should always show the same way as the previous rank. I don't add or delete the fixed amount of dataset that I use. Even though the same dataset is divided randomly, what I think is there shouldn't be a significant impact on the performance because the ratio of data used between training and validation is 80(training):20(validation). If they change their behaviors each time I run on the same unchanged fixed amount of data, how can I tell which one shows better performance on the current dataset? I understand the training data changes but its percentage is 80% of the dataset.

Improve this question
$\endgroup$
1
  • $\begingroup$ Do you really have 100k records? Because with so many records you should get relatively stable results. If you had fewer samples there could be different reasons why you're getting unstable results. Also how high are the performance fluctuations you are observing after each run? Are they like +/- 0.1%, +/- 1%, +/- 10%? $\endgroup$
    – stmax
    Commented Apr 13, 2017 at 12:38

2 Answers 2

Reset to default
2
$\begingroup$

How are you dividing? You say randomly, but is this being redone each time your analysis is run? If it is, then this is perfectly normal, expected behavior. You're getting a different prediction performance because you have a different random training set each time.

You could also be using a stochastic method, that doesn't give the same result each time, say regression fit using gradient descent versus least squares. Since you didn't give any more details, I can't be more specific than that.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I have added some more detail, please let me understand what I am thinking wrong. $\endgroup$
    – user122358
    Commented Apr 13, 2017 at 0:44
  • $\begingroup$ @user122358: I think this is still unclear: "You say randomly, but is this being redone each time your analysis is run?" Do you split your dataset once and then repeat training several times on the same split or do you repeat splitting + training several times? $\endgroup$
    – stmax
    Commented Apr 13, 2017 at 15:28
  • $\begingroup$ I have a fixed amount of data. Each time I train the classifiers, dataset is divided into training and validation randomly. But the dataset is the same. $\endgroup$
    – user122358
    Commented Apr 15, 2017 at 12:59
  • $\begingroup$ @user122358: The overall data may be the same, but the splitting into training and test set are probably being split randomly. Thus, you'd be using different cases to train and test each time. $\endgroup$
    – MentatOfDune
    Commented Apr 16, 2017 at 23:55
2
$\begingroup$

Depending on a particular implementation of a classifier it may have a random initiation seed. Try to look that up in the documentation and set it to a fixed value. E.g. random_state=42.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is likely the answer. You can make it truly random, or you can make it random with a seed so that the results are reproducible. $\endgroup$
    – CalZ
    Commented Apr 13, 2017 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.