I am working on titanic dataset, I achieved 92% accuracy using random forest. However, the accuracy score dropped to 89% after I tuned it using Gridsearch. Now, I was wondering if it caused by imbalanced dataset, since it has only 342 out of 891 passenger survived the disaster. Would appreciate the clarification.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ was the 92% accuracy on the train set or validation set or crossvalidation (or god forbid the test set)? and what about the 89%? $\endgroup$– A KareemCommented May 4, 2020 at 15:11
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Welcome! You haven't given us enough information to be able to diagnose this issue completely, but you should check your grid search code to see how each cross-validated model is being trained and note which parameters are different from those used with the 92% model.
If it has something to do with the unbalanced data, it's because you're not stratifying your sampling of the dataset with respect to the labels (making sure training and validation sets have the same ratio of classes (in this case, 38% survive, 62% don't survive.))
I would guess that something is going on with your cross validation process. If I had to guess specifics (and again, we can't say for sure given what you've posted here) , I would say that something you're doing in the CV probably results in those models not using as much training data as the 92% model.
-
$\begingroup$ To add to this, I'd also state that a 3% difference in a dataset that seems to only have 1000 samples is small enough that it's hard to state with certainty that something wrong is going wrong here and not just random noise. $\endgroup$– A KareemCommented May 4, 2020 at 15:18
-
$\begingroup$ Right on. The 92% could definitely be a lucky draw from a train-test split (or any number of other things, as you say). $\endgroup$– MatthewCommented May 4, 2020 at 15:31