1
$\begingroup$

Given a database, I split the data in train and test. I want to use a decision-tree classifier (sklearn) for a binary classification problem. Considering I already found the best parameters for my model, if I run the model on the test set I obtain at each run (considering the same hyper-parameters) a different result. Why that? Considering I am using as metric the accuracy score, I have variations from 0.5 to 0.8. Which result should I take as correct, because I am not sure If it's correct to take the best result on the test set, or maybe I should consider an average of the results... I dont know. Example, scoring for gridsearchCV is accuracy. I obtain Grid scores on development set:

0.627 (+/-0.129) for {'max_features': 2}
0.558 (+/-0.152) for {'max_features': 3}
--> Best parameters: {'max_features': 2}
    Best score: 0.626876876876877 (this is the accuracy)

Using the best estimator on the test set, I obtain accuracy 0.83.. which I think it is only due to the case. In fact I try again and the result is this

Grid scores on development set:
0.584 (+/-0.126) for {'max_features': 2}
0.572 (+/-0.168) for {'max_features': 3}
--> Best parameters: {'max_features': 2}
    Best score: 0.5840215215215215
On the test set accuracy: 0.62 !!

So, how can I trust my results? Second, would not it be better to use CV on all data, instead of just splitting at the beginning the dataset in test and train just one time?

I read about random_state but the problem is that the results depend on which value I use.. example random_state = 2 --> accuracy 0.6 (on test set) random_state = 6 --> accuracy 0.79 (on test set) so basically this does not resolve my problem. How can I validate my model if I dont know which one to use?

$\endgroup$
0
$\begingroup$

how can I trust of my results?

You probably shouldn't trust your results, because the large variation is likely caused by overfitting. Basically your model is not very reliable.

My guess is that you have either too many features or not enough instances, or any combination of these two issues.

I am not sure If it's correct to take the best result on the test set, or maybe I should consider an average of the results

It is definitely not correct to take the best result. The average result is more representative of the true performance, but you should also provide the standard deviation.

Second, would not it be better to use CV on all data, instead of just splitting at the beginning the dataset in test and train just one time?

Yes, that's a good idea: as far as I understand, currently you're just manually running the program several times and obtaining different performance each time. You can indeed use Cross-validation instead.

I read about random_state but the problem is that the results depend on which value I use.. example random_state = 2 --> accuracy 0.6 (on test set) random_state = 6 --> accuracy 0.79 (on test set) so basically this does not resolve my problem. How can I validate my model if I dont know which one to use?

That's normal: the split function separates the training and test set randomly according to a random sequence. Setting the random seed to a particular value guarantees the exact same sequence every time, but you don't want that since it makes the splitting non-random.

Instead you should work on reducing the variation. Usually it's not possible to add instances, so you should probably try to reduce the number of features, or simplify them.

| improve this answer | |
$\endgroup$
  • $\begingroup$ thank you so much for the answer. I think y are right, that the instability on the results is due to overfitting. Considering it s not possible to add more samples, I should reduce the number of features in input. This is something I already did, because at the beginning I had 70 features. I applied some method of features selection (LASSO or GBM). So, what do u think it is better, to apply again a feature selection on the 20 features, or consider the most important ones from the previous feature selection? $\endgroup$ – Mark Mar 20 at 12:55
  • $\begingroup$ @Mark it's difficult to know without knowing the data, it depends on many things. I suggest you experiment all the options you can think of, and then see if any of these options give more stable results. $\endgroup$ – Erwan Mar 20 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.